System and method of overlaying real-time data onto an autonomous vehicle routing cost model

ABSTRACT

A method includes of routing an autonomous vehicle includes receiving information obtained from a camera on a second vehicle distinct from the autonomous vehicle. The method includes automatically identifying a road condition using image analysis of the information received from the camera on the second vehicle. The method includes receiving a request to route the autonomous vehicle from a first location to a second location; and in response to the request: generating a cost model for routing the autonomous vehicle, wherein the cost model includes a cost of the road condition automatically identified from the information received from the camera on the second vehicle; selecting a route from the first location to the second location in accordance with the cost model; and routing an autonomous vehicle in accordance with the selected route.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/726,623, filed Dec. 24, 2019, which is a continuation of U.S.application Ser. No. 16/597,801, filed Oct. 9, 2019, entitled “SystemAnd Method For Routing Using Intersection Costs,” now U.S. Pat. No.10,563,993, issued on Feb. 18, 2020, which is a continuation of PCTApplication PCT/US18/56740, filed Oct. 19, 2018, entitled “AutonomousVehicle Routing,” which is a continuation of U.S. application Ser. No.16/164,708 filed Oct. 18, 2018, entitled “Autonomous Vehicle Routing,”which claims priority to U.S. Provisional Application No. 62/740,882filed Oct. 3, 2018, entitled “Autonomous Vehicle Routing”; U.S.Provisional Application No. 62/685,106 filed Jun. 14, 2018, entitled“Autonomous Vehicle Routing”; U.S. Provisional Application No.62/599,610 filed Dec. 15, 2017, entitled “Autonomous Vehicle Routing”;and U.S. Provisional Application No. 62/574,737 filed Oct. 19, 2017,entitled “Autonomous Vehicle Routing,” each of which is incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The disclosed embodiments relate generally to routing autonomousvehicles.

BACKGROUND

In the coming years, autonomous vehicle (AV) technology will overcomethe present challenges in motion planning and control. For example,autonomous vehicles will be able to stay in lanes, follow cars, avoidpedestrians and drive like a taxi driver patrolling the streets.Autonomous vehicles will need only to be told where to go and how to getthere, making route planning critical in the AV-driven world.

Thus, as developers build core autonomy technology and start to scaletheir fleets of self-driving vehicles, whether for sale to individualconsumers or for starting their own ride-sharing networks, they willneed effective routing technology. The winners and losers in this racewill be determined by which companies operate the most efficientnetworks with the highest vehicle utilization.

SUMMARY

(A1) A method of routing an autonomous vehicle is provided. The methodis performed at a computer system including one or more processors andmemory. The method includes generating a cost model for routing theautonomous vehicle. The cost model includes one or more costs other thantravel time, the one or more costs other than travel time selected fromthe group consisting of: a cost of traversing an area where autonomousdriving is prohibited, a cost of a driving maneuver, a cost of acharacteristic of the autonomous vehicle, a cost of a weather condition,a cost of a time of day, a cost of a sunlight angle, a cost of apedestrian traffic pattern, a cost of a non-motorized vehicle trafficpattern, a cost of a pavement condition, a cost of a road age, a cost ofan event present at the time of a routing request, a cost of a roadlacking lane lines, a cost of an electric-charging constraint, and acost of acceleration. The method further includes receiving a request toroute the autonomous vehicle from a first location to a second location.The method includes, in response to the request to route the autonomousvehicle, selecting a route from the first location to the secondlocation in accordance with the cost model and routing the autonomousvehicle along the selected route.

(A2) In some embodiments of (A1), the cost model is generated inresponse to the request to route the autonomous vehicle.

(A3) In some of the embodiments of any of (A1)-(A2), the cost model is afirst cost model for the autonomous vehicle. The method includesreceiving a request to route a non-autonomous vehicle from a thirdlocation to a fourth location. The method further includes generating asecond cost model for routing the non-autonomous vehicle. The secondcost model is distinct from the first cost model and does not includeone or more of the costs other than travel time that are included in thefirst cost model. The method further includes, in response to therequest to route the non-autonomous vehicle: selecting a route from thethird location to the fourth location in accordance with the second costmodel and routing the non-autonomous vehicle along the selected routefor the non-autonomous vehicle.

(A4) In some of the embodiments of any of (A1)-(A3), the autonomousvehicle is a first autonomous vehicle having first autonomous drivingcapabilities. The cost model is a first cost model for the firstautonomous vehicle. The one or more costs other than travel time for thefirst cost model include a cost corresponding to the first autonomousdriving capabilities. The method includes receiving a request to route asecond autonomous vehicle from a fifth location to a sixth location. Thesecond autonomous vehicle has second autonomous driving capabilitiesdifferent from the first autonomous driving capabilities. The methodfurther includes generating a third cost model for routing the secondautonomous vehicle. The third cost model is distinct from the first costmodel and includes a cost corresponding to the second autonomous drivingcapabilities. The method further includes, in response to the request toroute the second autonomous vehicle: selecting a route from the fifthlocation to the sixth location in accordance with the second cost modeland routing the second autonomous vehicle along the selected route forthe second autonomous vehicle.

(A5) In some of the embodiments of any of (A1)-(A4), the cost modelfurther includes a travel-time cost and the route is selected inaccordance with both the travel-time cost and the one or more costsother than travel time.

(A6) In some of the embodiments of any of (A1)-(A5), the one or morecosts other than travel time include the cost of an event present at thetime of the routing request and the event present at the time of therouting request is a construction project or a traffic accident.

(A7) In some of the embodiments of any of (A1)-(A6), the one or morecosts other than travel time include the cost of an event present at thetime of the routing request, the event present at the time of therouting request is a regularly-scheduled event, and the selecting theroute is performed based at least in part on the cost of theregularly-scheduled event, in accordance with a determination that theregularly-scheduled event is occurring at the time of the routingrequest.

(A8) In some of the embodiments of any of (A1)-(A6), the one or morecosts other than travel time include the cost of an event present at thetime of the routing request, the event present at the time of therouting request is a regularly-scheduled event, and selecting the routeis performed based at least in part on the cost of theregularly-scheduled event, in accordance with a determination that theregularly-scheduled event will be occurring at a time when theautonomous vehicle is expected to reach the regularly-scheduled event.

(A9) In some of the embodiments of any of (A1)-(A8), the drivingmaneuver is selected from the group consisting of a lane change, a rightturn, a left turn, an unprotected left turn, a U-turn, and a highwaymerge.

(A10) In some of the embodiments of any of (A1)-(A9), the one or morecosts other than travel time include the cost of a characteristic of theautonomous vehicle and the characteristic of the autonomous vehicle is avehicle type.

(A11) In some of the embodiments of any of (A1)-(A10), the one or morecosts other than travel time include the cost of a characteristic of theautonomous vehicle and the characteristic of the autonomous vehicle isan autonomous driving capability of the autonomous vehicle.

(A12) In some of the embodiments of any of (A1)-(A11), at least one ofthe one or more costs other than travel time is a cost for a particulargeographical area, road, lane within a road, or maneuver.

(A13) In some of the embodiments of any of (A1)-(A12), the one or morecosts other than travel time include, for a plurality of differentroads, a plurality of distinct road-specific costs for a shared weathercondition at the different roads.

(A14) In some of the embodiments of any of (A1)-(A13), the one or morecosts other than travel time include, for a plurality of differentroads, a plurality of distinct road-specific costs for different weatherconditions at the different roads.

(A15) In some of the embodiments of any of (A1)-(A14), the one or morecosts other than travel time include a plurality of distinctroad-specific costs of the time of day for different roads.

(A16) In some of the embodiments of any of (A1)-(A15), the one or morecosts other than travel time include a plurality of distinctroad-specific costs of the sunlight angle for different roads.

(B1) A method of routing an autonomous vehicle is provided. The methodis performed at a computer system including one or more processors andmemory. The method includes receiving a request to route the autonomousvehicle from a first location to a second location, identifying a set ofone or more autonomous driving capabilities of the autonomous vehicle,selecting a route from the first location to the second location inaccordance with the set of one or more autonomous driving capabilitiesof the autonomous vehicle, and routing the autonomous vehicle inaccordance with the selected route.

(B2) In some of the embodiments of (B1), the autonomous vehicle is afirst autonomous vehicle. The method includes receiving a request toroute a second autonomous vehicle from a third location to a fourthlocation. The method further includes identifying a set of one or moreautonomous driving capabilities of the second autonomous vehicle,wherein the set of one or more autonomous driving capabilities of thesecond autonomous vehicle include at least one autonomous drivingcapability different from the set of one or more autonomous drivingcapabilities of the first autonomous vehicle. The method furtherincludes selecting a route from the third location to the fourthlocation in accordance with the set of one or more autonomous drivingcapabilities of the second autonomous vehicle and routing the secondautonomous vehicle in accordance with the selected route for the secondautonomous vehicle.

(B3) In some of the embodiments of any of (B1)-(B2), identifying the setof one or more autonomous driving capabilities of the autonomous vehicleincludes receiving an autonomous driving capability with the request.

(B4) In some of the embodiments of any of (B1)-(B3), the method furtherincludes, at the computer system, receiving, with the request to routethe autonomous vehicle, an identifier for the autonomous vehicle. Theidentifying comprises looking up one or more autonomous drivingcapabilities of the autonomous vehicle based on the identifier.

(B5) In some of the embodiments of any of (B1)-(B4), the method furtherincludes, at the computer system, generating a cost model for routingthe autonomous vehicle that includes one or more costs for the set ofone or more autonomous driving capabilities of the autonomous vehicle.The selecting comprises selecting the route from the first location tothe second location based at least in part on the cost model.

(B6) In some of the embodiments of any of (B1)-(B5), the set of one ormore autonomous driving capabilities of the autonomous vehicle includesa capability of sensors of the autonomous vehicle.

(B7) In some of the embodiments of any of (B1)-(B6), the set of one ormore autonomous driving capabilities of the autonomous vehicle includesa maneuverability of the autonomous vehicle.

(B8) In some of the embodiments of any of (B1)-(B7), the set of one ormore autonomous driving capabilities of the autonomous vehicle includesa limitation on the autonomous vehicle determined from historicalperformance data for at least one of the autonomous vehicle and vehiclesof a same type as the autonomous vehicle.

(B9) In some of the embodiments of any of (B1)-(B8), the set of one ormore autonomous driving capabilities of the autonomous vehicle includesa safety rating for the autonomous vehicle and the selecting isperformed based at least in part on the safety rating of the autonomousvehicle.

(B10) In some of the embodiments of any of (B1)-(B9), the set of one ormore autonomous driving capabilities of the autonomous vehicle includesa plurality of autonomous driving capabilities of the autonomous vehicleand the selecting is performed based at least in part on the pluralityof autonomous driving capabilities of the autonomous vehicle.

(C1) A method of routing an autonomous vehicle is provided. The methodis performed at a computer system including one or more processors andmemory. The method includes generating a cost model for routing theautonomous vehicle. The cost model includes, for an intersection, aplurality of costs for traversing distinct paths through theintersection. The method includes receiving a request to route theautonomous vehicle from a first location to a second location. Themethod includes, in response to the request to route the autonomousvehicle, selecting a route from the first location to the secondlocation in accordance with the cost model. The selecting is based atleast in part on one of the plurality of costs for traversing thedistinct paths through the intersection. The method includes routing theautonomous vehicle in accordance with the selected route.

(C2) In some of the embodiments of (C1), the cost model is representedas a graph of nodes and edges, the respective edges having respectiveedge weights that represent costs. In some embodiments, the methodfurther comprises, at the computer system, representing the intersectionas a plurality of nodes and a plurality of edges having edge weights.Each edge of the representation of the intersection represents adistinct path through the intersection.

(C3) In some of the embodiments of (C2), the method further includes, atthe computer system, forgoing representation of a forbidden path throughthe intersection.

(C4) In some of the embodiments of any of (C1)-(C3), the intersection isan intersection of a plurality of roads, each road in the plurality ofroads having one or more costs distinct from the plurality of costs fortraversing the distinct paths through the intersection.

(C5) In some of the embodiments of any of (C1)-(C4), the plurality ofcosts for traversing the distinct paths through the intersection includeone or more costs of a driving maneuver for traversing one of thedistinct paths through the intersection.

(C6) In some of the embodiments of (C5), the driving maneuver isselected from the group consisting of a lane change, a right turn, aleft turn, an unprotected left turn, a U-turn, and a highway merge.

(D1) A method of routing an autonomous vehicle is provided. The methodis performed at a computer system including one or more processors andmemory. The method includes receiving information obtained from a cameraon a second vehicle distinct from the autonomous vehicle, automaticallyidentifying a road condition using image analysis of the informationreceived from the camera on the second vehicle and receiving a requestto route the autonomous vehicle from a first location to a secondlocation. The method further includes, in response to the request,generating a cost model for routing the autonomous vehicle. The costmodel includes a cost of the road condition automatically identifiedfrom the information received from the camera on the second vehicle. Themethod includes selecting a route from the first location to the secondlocation in accordance with the cost model and routing an autonomousvehicle in accordance with the selected route.

(D2) In some of the embodiments of (D1), generating the cost modelincludes updating a previous cost model in response to the request.

(D3) In some of the embodiments of any of (D1)-(D2), the methodincludes, at the computer system, determining whether the road conditionis present at the time of the request. In some embodiments, the cost ofthe road condition is included in the cost model in accordance with adetermination that the road condition is present at the time of therequest.

(D4) In some of the embodiments of any of (D1)-(D3), the road conditionautomatically identified from the information received from the cameraon the second vehicle is a condition selected from the group consistingof: a weather condition, a road condition due to weather, a roadcondition due to construction, and a road condition due to a trafficaccident.

(D5) In some of the embodiments of any of (D1)-(D3), the road conditionautomatically identified from the information received from the cameraon the second vehicle is a condition selected from the group consistingof a sunlight angle, pedestrian traffic, non-motorized vehicle traffic,a pavement condition, and a lack of lane lines.

(D6) In some of the embodiments of any of (D1)-(D5), the camera is adashcam.

(E1) A routing method for a ride-share transportation network isprovided. The method includes storing representations of a plurality offirst passengers. Each of the representations of the plurality of firstpassengers includes a requested pick-up location and a requesteddrop-off location for a respective first passenger of the plurality offirst passengers. The method further includes storing representations ofa plurality of fleet vehicles. The method further includes generating agraph representation of a geographic map that includes the requestedpick-up locations and drop-off locations for the plurality of firstpassengers. The method further includes generating a state graphrepresentation of the plurality of first passengers and the fleetvehicles. The state graph representation includes a plurality of nodesconnected by edges. Each node of the plurality of nodes of the stategraph representation represents a candidate state of the plurality offirst passengers and the fleet vehicles. A respective edge of the stategraph representation represents an action of a respective vehiclepicking up or dropping off a passenger. The respective edge has a costthat is based at least in part on traversal of the graph representationof the geographic map. The method further includes generating a set ofroutes for the fleet vehicles, including assigning each passenger of theplurality of first passengers to be picked up and dropped off by arespective vehicle of the fleet vehicles. The routes for the fleetvehicles are generated by performing a graph search of the state graphrepresentation by evaluating a cost model that includes the costs of therespective edges. The method further includes routing the fleet vehiclesin accordance with the generated set of routes.

(E2) In some of the embodiments of (E1), the fleet vehicles include aplurality of autonomous vehicles.

(E3) In some of the embodiments of any of (E1)-(E2), the plurality offleet vehicles is operated by a single operator.

(E4) In some of the embodiments of any of (E1)-(E3), the method includesreceiving a ride request from a second passenger, distinct from theplurality of first passengers, wherein the request includes a requestedpick-up location and a requested drop-off location for the secondpassenger. The method further includes updating the route for arespective vehicle of the fleet vehicles, including assigning the secondpassenger to be picked up and dropped off by the respective vehicle, inaccordance with one or more least-expensive-insertion criteria.

(E5) In some of the embodiments of (E4), updating the route for therespective vehicle of the fleet vehicles includes inserting a pick-uplocation and a drop-off location for the second passenger into anexisting route for the respective vehicle without modifying pick-up anddrop-off locations for passengers already assigned to the respectivevehicle.

(E6) In some of the embodiments of (E4), updating the route for therespective vehicle of the fleet vehicles includes: inserting a pick-uplocation and a drop-off location for the second passenger into anexisting route for the respective vehicle and reassigning a passengeralready assigned to the respective vehicle to a different vehicle of thefleet vehicles.

(E7) In some of the embodiments of any of (E1)-(E6), generating the setof routes for the fleet vehicles includes: assigning each passenger ofthe first plurality of passengers to one or more candidate pick-uplocations based on the first passenger's requested pick-up location;assigning each passenger of the first plurality of passengers to one ormore candidate drop-off locations based on the first passenger'srequested drop-off location; clustering the first plurality ofpassengers according to their assigned candidate pick-up locations anddrop-off locations; assigning respective vehicles of the fleet vehiclesto a plurality of clusters; and parallelizing the routing of the fleetvehicles according the assigned plurality of clusters for the respectivevehicles of the fleet vehicles.

(E8) In some of the embodiments of (E7), thee method further includesreceiving a ride request from a second passenger, distinct from theplurality of first passengers. The request includes a requested pick-uplocation and a requested drop-off location for the second passenger. Themethod further includes updating the route for a respective vehicle ofthe fleet vehicles, including assigning the second passenger to bepicked up and dropped off by the respective vehicle, by assigning thesecond passenger to an existing cluster and updating the route for thevehicle assigned to the existing cluster.

(E9) In some of the embodiments of any of (E1)-(E8), performing thegraph search includes limiting a number of states explored in the stategraph representation using a lowest-bound heuristic by forgoingsubsequent exploration from any nodes in the state graph representationfor which a lowest-bound of the cost function exceeds a cost of analready-determined set of routes for the fleet vehicles.

(E10) In some of the embodiments of any of (E1)-(E9), the cost modelincludes pick-up wait times for the first passengers.

(E11) In some of the embodiments of any of (E1)-(E10), the graph searchis a bi-directional search.

Some embodiments of the present disclosure provide a computer system(e.g., a server system), comprising one or more processors and memorystoring one or more programs. The one or more programs storeinstructions that, when executed by the one or more processors, causethe computer system to perform any of the methods described here

Some embodiments of the present disclosure provide an autonomousvehicle. The autonomous vehicle includes a computer system, comprisingone or more processors and memory storing one or more programs. The oneor more programs store instructions that, when executed by the one ormore processors, cause the one or more processors to perform any of themethods described herein.

Some embodiments of the present disclosure provide a non-transitorycomputer readable storage medium storing instructions that, whenexecuted by a computer system having one or more processors, cause thecomputer system to perform any of the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The embodiments disclosed herein are illustrated by way of example, andnot by way of limitation, in the figures of the accompanying drawings.Like reference numerals refer to corresponding parts throughout thedrawings.

FIG. 1 is a two-dimensional map showing an example of a conventional(non-autonomous vehicle) route.

FIG. 2 is a two-dimensional map showing an example of an autonomousvehicle route, in accordance with some embodiments.

FIGS. 3A-3B is a block diagram illustrating an architecture of anautonomous vehicle routing engine, in accordance with some embodiments.

FIG. 4A is a two-dimensional birds-eye view illustrating an example ofan intersection that is represented as a single node, in accordance withsome embodiments.

FIG. 4B is a two-dimensional birds-eye view illustrating an example ofan intersection that is represented as a plurality of nodes (instead ofa single node) and a plurality of edges, where some of the edgesrepresent paths through the intersection, in accordance with someembodiments.

FIG. 5 is a two-dimensional birds-eye view illustrating an example of aroute feature in which lane-level details and constraints are useful forproviding drivable routes for autonomous vehicles, in accordance withsome embodiments.

FIG. 6 is a two-dimensional birds-eye view illustrating another examplewhere paths through an intersection are represented with their own edgesin a graph, in accordance with some embodiments.

FIG. 7 is a two-dimensional birds-eye view illustrating an example of aroute that heavily penalizes human-intervention-zone transitions, inaccordance with some embodiments.

FIG. 8 is a two-dimensional map illustrating a conventional example of arouting graph where roads are edges and intersections are nodes.

FIG. 9 is saturation histograms for a road under sunny and snowyconditions (shown in photographs also included in FIG. 9 ). Thesaturation histograms are an example of an analysis that can be used toautomatically determine road conditions and assign costs or attributesto roads that are then used in autonomous vehicle routing, in accordancewith some embodiments.

FIG. 10 is a visualization of a boosted cascaded Haar classifier, inaccordance with some embodiments.

FIG. 11 is block diagram screen shot of a cascaded LBP classifier, inaccordance with some embodiments.

FIG. 12 is a block diagram of a “You Only Look Once” (YOLO) architecturefor real-time object detection, in accordance with some embodiments.

FIG. 13 is a series of photographs illustrating detection of varioustraffic signs using a convolutional neural net model, in accordance withsome embodiments.

FIG. 14 is a block diagram illustrating a client-server environment, inaccordance with some embodiments.

FIGS. 15A-15B are a flowchart of a vehicle routing method using a costmodel with costs other than travel time, in accordance with someembodiments.

FIG. 16 is a flowchart of a method of autonomous-vehicle routing using acost model with vehicle constraints, in accordance with someembodiments.

FIG. 17 is a flowchart of a method of autonomous-vehicle routing using acost model with intersection costs, in accordance with some embodiments.

FIG. 18 is a flowchart of a method of overlaying real-time data onto anautonomous-vehicle-routing cost model, in accordance with someembodiments.

FIG. 19 is a block diagram illustrating an example architecture forrouting a fleet of vehicles, in accordance with some embodiments.

FIG. 20 illustrates stops assigned to requested pick-up and drop-offlocations, in accordance with some embodiments.

FIG. 21 illustrates ride request graphs for ungrouped and grouped riderequests, in accordance with some embodiments.

FIG. 22 is a two-dimensional spatial representation illustratingcreation of cluster pairs of vehicles, in accordance with someembodiments.

FIG. 23 is a two-dimensional map a combined (e.g., unioned) route for aplurality of vehicles, in accordance with some embodiments.

FIG. 24 is a state graph representation of a plurality of passengers andvehicles, in accordance with some embodiments.

FIGS. 25A-25B are two-dimensional spatial representations illustratingthe addition of an insertion point to an existing route, in accordancewith some embodiments.

FIGS. 26A-26C are a flowchart of a routing method for a ride-sharetransportation network, in accordance with some embodiments.

DETAILED DESCRIPTION

Autonomous vehicles are not able to robustly handle every situation theymay encounter on the road. To address this problem, the presentdisclosure provides routing devices and methods that take into accountAV-focused requirements and constraints (e.g., requirements andconstraints that are particularly, although possibly not exclusively,important to autonomous vehicles). These methods and devices improveautonomous vehicles by avoiding situations they cannot handle or mayhave difficulty handling. By allowing autonomous vehicles to avoid suchsituations, the routing devices and methods described herein makeautonomous vehicles safer for passengers and pedestrians, perhaps evensaving lives, and may make riding in autonomous vehicles morecomfortable as well.

The present disclosure provides methods and devices that use cost modelsin vehicle routing (e.g., autonomous vehicle routing). As used herein,the term “cost model,” in the context of vehicle routing, means anassignment or schedule of “costs” associated with aspects of a trip(i.e., the trip being routed). The costs in the cost model need not be,and generally are not, financial costs. For example, in accordance withsome embodiments, the cost models provided herein use travel time costs,costs of a passenger waiting to be picked-up, safety costs (e.g., costsof an autonomous vehicle making an unprotected left turn), and manyothers. In some embodiments, the overall cost of a route is the sum ofall of the individual costs of that route (e.g., the sum of all of costsof respective aspects of the route for which costs have been assigned).Generally speaking, routes with lower costs are preferentially selectedover routes with higher costs. Thus, for example, assigning a highercost to an unprotected left turn means that the systems and methodsprovided herein will generate routes with fewer unprotected left turns.As another example, assigning a higher cost to a passenger waiting to bepicked-up will result in shorter wait times for passengers to be pickedup, even if that means a longer amount of time spent in the vehicle orhaving the vehicle make more unprotected left turns. Stated another way,the cost model described herein use numerical costs to weight relativepreferences of various aspects of a potential routes.

To that end, some embodiments provide autonomous-vehicle (AV)-focusedmethods and devices that account for (e.g., optimize for) costs otherthan time and distance, taking into account, for example, differentweights for human-intervention zone transitions, weather, and pedestrianzones. Also, the operation of autonomous vehicles benefits from highdefinition (HD) and lane-level maps. Such HD map coverage will, however,increase incrementally. The present disclosure bridges the gap byproviding routing methods and devices that provide hybrid routes where asafety driver can take over in a regular road network and allow theautonomous vehicle to drive itself when HD map data is available.

For example, a conventional (non-AV) routing method might create theroute shown in FIG. 1 .

From an autonomous vehicle standpoint, there are numerous problems withthis route. The route does not consider which maneuvers are safe for anautonomous vehicle or consider AV-safe pickup and drop-off points. Forexample, the route 100 (e.g., the bolded dashed lines) includes drop-offpoint 102, which is not considered an AV-safe drop-off point. The route100 also includes intersection 104. The intersection 104 creates anillegal AV maneuver point (e.g., to make a left turn). Thus, the routeis not customized based on the constraints of a particular autonomousvehicle. The route does nothing to prefer autonomous-vehicle zones(e.g., avoid human-only zones), reduce the number of human interventionsthat are necessary, or maximize the amount of time that the vehicleoperates autonomously. For example, the route 100 also includes a humanintervention zone 106.

Additionally, a route generated from a conventional routing engine isnot usually represented in a format that autonomous vehicles can use(e.g., a route geometry is typically along the center of the road). Forexample, in a bidirectional road, the centerline goes along the laneline dividing opposing traffic. Because autonomous vehicles drive inlanes, they may need their routes defined in terms of those lanes.Techniques that attempt to “map match” a road-centered route to alane-centered map are prone to matching error, especially atintersections, where the road geometry often does not correspond to thelane geometry.

Furthermore, depending upon the level of autonomy for the autonomousvehicle, an AV-route may or may not need more details specific to themotion planner. Finally, the route shown above does not support bothAV-level datasets (e.g., with lane data) and road-level data sets. Therouting methods and devices described herein may take these issues intoaccount when routing an autonomous vehicle. For example, the route 200(e.g., represented by the bold dashed lines) shown in FIG. 2 may begenerated in accordance with embodiments described herein. The route 200shown in FIG. 2 considers which maneuvers are safe for an autonomousvehicle (e.g., avoids intersection 104), considers AV-safe pickup anddrop-off points, (e.g., avoids unsafe drop-off point 102) and considershuman-intervention zones (e.g., avoids human-intervention zone 106). Aroute such as the route 200 shown in FIG. 2 may be customized based onthe constraints of a particular autonomous vehicle. The routing methodsand devices described herein can handle both human andautonomous-vehicle requirements. For example, these methods and devicesinclude some or all of the following features:

-   -   Both lane-level and road-level maps;    -   Consideration of both road-level traffic and lane-level traffic        where available;    -   Consideration of autonomous vehicle constraints (e.g.,        assignment of costs for autonomous vehicle constraints in a cost        model), including constraints specific to an individual vehicle        or request (e.g., a cost due to the fact that a specific make        and model can only autonomously travel on certain types of        roads, such as bidirectional highways with no dividers) as well        as constraints for autonomous vehicles generally (e.g., the        methods and devices choose AV-focused routes that avoid        blacklisted areas and reduce/minimize human intervention);    -   Improvement/optimization for different objectives, e.g., fastest        time, maximum autonomy time;    -   Flexibility to change costs in real-time;    -   Consideration of optimal pickup/drop-off locations for        autonomous vehicles;    -   Consideration of charging constraints;    -   Uniform SDK for offline routing (e.g., in-vehicle or onboard)        and online routing (e.g., in the cloud);    -   A focus on safety in addition to efficiency;    -   Scalability with a flexible architecture; and    -   Online and offline modes, the latter for when autonomous        vehicles lose connectivity.

Table 1 highlights some of the differences between AV-routing andconventional routing in accordance with some embodiments.

TABLE 1 Conventional Routing AV-Routing Algorithm Fastest-pathalgorithms Flexible-path algorithms with custom constraints Map DataRoad-level data only Hybrid routing across HD Meter-level accuracy mapdata and road-level data Handling of any vendor's or customer's dataCentimeter-level accuracy Dynamic Data Historical and real-timeHistorical and real-time traffic traffic Human Input for Machine Inputvia video incidents (e.g., computer vision algorithms) Online/OfflineOnline taking advantage Online and offline using of fresh data sameengine and Offline - limited quality, producing consistent result butalways available

Fundamentally, routing relies on 3 components: a graph of nodes andedges, edge weights, and algorithms to traverse the graph. The graphrepresents the traversable space of lanes and/or roads that theautonomous vehicle can travel. The edge weights are the costs oftraversing each road/lane. Algorithms such as Dijkstra, A*, andBidirectional Dijkstra explore the weighted graph to find paths whichminimize a cost function between an origin and destination.

In some embodiments, the routing methods and devices make use of ahybrid map, built up from a conventional road map as well as alane-level map. A base-layer road map can be sourced, for example, fromOpen Street Map (OSM). Lane-level maps can be obtained from OEM's andother 3rd-party providers or can be created from sensor data (e.g.,autonomous vehicle sensor data or dashcam data from human-drivenvehicles).

From the basic road map, a static road graph of default edge weights isconstructed (e.g., accounting for human intervention zones, speedlimits, autonomous vehicle turn restrictions, etc.). During run-time, areal-time data layer is kept in memory, which listens for updates fromvarious real-time data pipelines. For each request, a new cost function(e.g., cost model) may be created and used to traverse the graph.

FIG. 3A is an example of an architecture of an autonomous-vehiclerouting engine, in accordance with some embodiments. For example, theclient 330 is the autonomous vehicle to be routed or an electronicdevice associated with the autonomous vehicle.

Real-time Data updates 340 includes a server system that receives and/ortracks real-time traffic 342, historical traffic 344, and AV Events 346and processes/forwards the traffic and events to AV Routing Engine 338,such that AV Routing Engine 338 can create/update the route for client330. The AV Routing Engine 338 also uses information received fromhybrid map 336 (e.g., which is based on road level map 332 and lanelevel map 334) to create/update the route for client 330.

FIG. 3B illustrates another exemplary architecture (e.g., a so-called“stack”) for a fleet of vehicles. The features of the exemplaryarchitecture shown in FIG. 3B may optionally compliment, replace, or becombined with the features of the architecture described with respect toFIG. 3A. In some embodiments, the fleet of vehicles is a mixed fleet ofvehicles including autonomous vehicles (e.g., autonomous vehicles 308)and non-autonomous vehicles (e.g., non-autonomous vehicles 306). In somecircumstances, a fleet includes a mix of different vehicle types (e.g.,automobiles, bicycles, scooters, and/or delivery robots). In somecircumstances, the fleet provides services to riders (e.g.,riders/consumers 304) by transporting riders from a first location to asecond location. In some circumstances, the fleet provides services toother consumers, e.g., by delivering items to the consumers (e.g.,delivering meals from restaurants, delivering packages from retailstores).

To facilitate the provision of these services using a mixed fleet ofvehicles, the stack includes a first server system 300 that providesfleet management services and routing information. The first serversystem 300 includes one or more processors (e.g., CPUs) and memorystoring instructions for the modules described with reference to thefirst server system (e.g., the map matching/positioning module 316, therouting engine 310, the geospatial silo-ed databases 312, and thenormalizing data schema 314). The first server system 300 interfaceswith a fleet manager 303 on a second server system 302. In the exemplaryarchitecture shown in the figure, the second server system 302 acts as aclient of the first server system 300, while riders, consumers (e.g.,riders/consumers 304), and vehicles (e.g., non-autonomous vehicles 306and/or autonomous vehicles 308) act as clients of the second serversystem 302.

In some embodiments, the second server system 302 is a separate anddistinct server system from the first server system 300. The secondserver system 302 includes one or more processors (e.g., CPUs) andmemory storing instructions for the modules described with reference tothe second server system 302 (e.g., the fleet manager 303). Theinstructions are executed by the one or more processors. In somecircumstances, the fleet manager 303 is one of a plurality of fleetmanagers serviced by the first server system 300. For example, the fleetmanager 303 may be a fleet manager for a specific company's fleet, andthe first server system 300 may provide services to multiple companies'fleets.

The first server system 300 includes a routing engine 310 that providesroutes and estimated times of arrival for autonomous vehicles 308 andnon-autonomous vehicles 306 in accordance with the various routingmethods described herein (e.g., as described with reference to FIG. 17and others). In some embodiments, a different instance of the routingengine is instantiated for each fleet.

The first server system includes one or more geospatial silo-eddatabases 312 storing geospatial data (e.g., data stored with acorresponding geographical location, such ai latitude and longitude).The geospatial silo-ed databases 312 include “standard definition” (SD)map data received from SD map data providers (e.g., data such as streetslocations/geometries, street names). An example of an SD Map DataProvider is OpenStreetMap. In some embodiments, the geospatial datafurther includes “high definition” data such as lane-level data (e.g.,lane locations/geometries, information about which maneuvers arepermitted from which lanes) received from HD map data providers. Thegeospatial data further includes data from other data providers, such asinformation received from municipalities about construction and roadclosures, real-time data (acquired as described above with reference toFIG. 18 ), traffic data and other data. In addition, the geospatialsilo-ed databases 312 store locations (e.g., map matched locations) ofthe vehicles in the various fleets.

In some embodiments, the geospatial silo-ed databases 312 store aplurality of distinct instances of data covering the same geographicalregion. For example, the geospatial silo-ed databases 312 store multiplemaps (e.g., with geospatial data overlaid on those maps) covering thesame region. In some circumstances, the different maps will include dataappropriate to a specific fleet's vehicles (e.g., a fleet will includeautonomous vehicles and delivery bots, and the geospatial silo-eddatabases will store one or more maps with information appropriate tothe fleet's vehicle types). Some instances of the map may be public toany client (e.g., any fleet manager), while other versions of the mapmay be private to certain clients (e.g., certain fleet managers). Forexample, a respective fleet may license data from a respective HD mapdata provider. The data provided by the respective HD map data providerare thus silo-ed and private to the respective fleet's fleet manager(e.g., fleet manager 303).

In some embodiments, the data ingested from the various data sources(e.g., the SD map data providers 318, the HD map data providers 320, theother data providers 322) is normalized by a normalizing data schema314. The normalizing data schema 314 translates data from a plurality offirst formats to a normalized second format that is independent of thefirst format (e.g., independent of the source of the data).

The first server system 300 further includes a map matching/positioningmodule 316 that matches location data received from vehicles to a maplocation (e.g., a location of a map stored in the geospatial silo-eddatabases 312). For example, some vehicles generate location data (e.g.,GPS data or data from another positioning system, such as GLONASS,Galileo, or BeiDou). In some circumstances, this data is noisy and mayplace the vehicle away from its actual location, e.g., on a sidewalk orin a building. The map matching/positioning module 316 receives thelocation data from a respective vehicle (e.g., through the fleet manager303, which interfaces with the first server system 300), matches thenoisy location data to a probable road location and/or lane location andupdates the “map matched” location of the vehicle in the geospatialsilo-ed databases 312 (e.g., updates the matched position). In addition,the map matched position is provided to the fleet manager 303 forvarious purposes (e.g., monitoring the fleet).

As noted above, the stack includes a second server system 302,optionally distinct and separate from the first server system 300. Thesecond server system 302 includes the fleet manager 303, which acts as aclient of the first server system 300 (e.g., a client of the routingengine). The fleet manager 303 dispatches vehicles (e.g., non-autonomousvehicles 306 and/or autonomous vehicles 308), assigns routes tovehicles, and assigns staging locations to vehicles within itscorresponding fleet (e.g., using information and routes provided by therouting engine). In addition, the fleet manager 303 interfaces withriders/consumers 304 (e.g., via a mobile application on the rider'ssmartphone or other device). The fleet manager 303 provides informationsuch as estimated times of arrival (ETAs) and trip costs to theriders/consumers 304 via their mobile devices. In some embodiments, thefleet manager 303 also receives data such as vehicle positions (e.g.,location, including optionally lane-specific location and orientation(e.g., which way the vehicle is pointing)) from the individual vehicles.

In some embodiments, an autonomous vehicle includes an AV platform whichserves as an operating system and framework for the autonomousfunctionality of the autonomous vehicle. The autonomous vehicle includesone or more processors (e.g., CPUs) and memory storing instructions forthe modules described with reference to the autonomous vehicle (e.g.,the interface module, the AV platform, drivers for thesensors/controls). The instructions are executed by the one or moreprocessors. An example of an AV platform is the open source RoboticsOperating System. The fleet manager (e.g., fleet manager 303) interactswith the autonomous vehicles (e.g., autonomous vehicles 308) through aninterface module, which is a module that runs on the AV platform toprovide routes to the AV platform and receive data from the autonomousvehicle's sensors. For example, in some circumstances, the interfacemodule is provided by the developer of the routing engine to act as aliaison between the first server system and the robotics of theautonomous vehicle. The AV platform receives sensor data from theautonomous vehicles sensor's and, in some circumstances, makes thesensor data available to the fleet manager, which can make the sensordata available further down the stack, for example, to the mapmatching/position module. In some embodiments, the AV platform sendscommands to the autonomous vehicle's controls (e.g., accelerationcommands, breaking commands, turning commands, etc.) through adrive-by-wire system.

Constructing the Routing Graph

In order to support the flexible cost scenarios above, a representationof the routing graph is constructed that can handle turn costs andlane-transition costs. In some embodiments, cost models include, for anintersection, costs for traversing different paths through theintersection (e.g., a path straight through the intersection, a leftturn at the intersection, a right turn at the intersection, or a U-turnat the intersection). To do so, the intersection may be represented as aplurality of nodes (instead of a single node 400, as shown in FIG. 4A)and a plurality of edges, where some edges represent respective pathsthrough the intersection. For example, a left turn is represented asnodes 402 and 404 connected by a first edge, which has a first weight. Apath straight through the intersection is represented by nodes 402 and406 connected by a second edge, which has a second weight (e.g., adifferent weight than the first weight). Illegal paths through theintersection (e.g., turning right to go the wrong way down a one-waystreet) are not represented in the graph, and thus cannot be explored bythe routing process. For example, there is no edge shown between node402 and 408 (e.g., a right turn).

Similarly, lane transition edges are added between lanes when necessary,which can capture the cost for the autonomous vehicle to change lanes.For example, lane transition edges are added anytime there is a changein the number of lanes in the road map. FIG. 5 illustrates an example ofa lane transition. A first road 500 in the road map of FIG. 5 is asingle lane. A second road 502 includes two lanes (e.g., a double lane).There are real-world situations where lane-level details and constraintsare useful to provide drivable (e.g., optimal) routes for autonomousvehicles as shown in the example in FIG. 5 .

In some embodiments, rather than being associated directly with weights,graph edges are associated with tags or attributes (e.g., the edge istagged as a human-intervention zone). A cost function evaluates theactual cost of each edge (e.g., in real-time). This allows certainaspects of the graph to remain static (e.g., the possible paths throughan intersection) while allowing the costs associated with those aspectsof the graph to be dynamic (e.g., an autonomous vehicle can request aroute with a particularly strong preference against human-interventionzones).

Normally, edges in a road graph are represented as either unidirectionalor bidirectional. This is prevalent in the industry and most routingengines use complex representations to handle edges based on theirdirectionality. In order to more simply represent the graph, however,some embodiments eliminate bidirectional edges in the graph. All edgesare thus unidirectional, and are aligned with their geometry. Edges mayfall into one of two categories, “original edges” or “transition edges.”Transition edges are created when expanding the original representation,and can capture turn/transition costs between specific original edges.This expansion helps capture turn costs correctly and provide low-cost(e.g., optimal) paths. An illustrative example is shown in FIG. 6 .

If a route is generated on the original graph 600 (on the left of FIG. 6) from the origin 602 to the destination 604, the turn costs may beadded once a given node is already being explored (i.e., once the nodehas been marked for resolution in Dijkstra's shortest-path algorithm).In this case, it is entirely possible to mark the node between c and efor resolution even before seeing whether coming from d is cheaper.

In this example, a path a→c→e has a total cost:

C(a→c→e)=C(a)+T(a→c)+C(c)+T(c→e)+C(e)

The second path has the following cost:

C(a→b→d→e)=C(a)+T(a→b)+C(b)+T(b→d)+C(d)+T(d→e)+C(e)

Note that the second path can have a smaller cost if:

C(a→b→d→e)<C(a→c→e)

C(a)+T(a→b)+C(b)+T(b→d)+C(d)+T(d→e)+C(e)<C(a)+T(a→c)+C(c)+T(c→e)+C(e)

C(b)+C(d)+T(a→b)+T(b→d)+T(d→e)<C(c)+T(a→c)+T(c→e)

T(c→e)>C(b)+C(d)−C(c)+T(a→b)+T(b→d)+T(d→e)−T(a→c)

This shows that one can arbitrarily raise the turn cost from c→e andfind suboptimal paths in the original representation. In the expandedgraph 606, however, because added extra nodes have been added to handleextra state, one can find optimal paths when considering transitioncosts between two edges. This situation can arise in cases where atransition is characterized as extremely difficult to make (e.g.,unprotected left turns for autonomous vehicles).

To fully characterize this expanded graph 606, the total numbers ofnodes and edges is significantly increased. In the example provided inFIG. 6 there is an increase in nodes from 3 nodes (in graph 600) to 8nodes (in expanded graph 606), and total edges from 5 edges (in graph600) to 10 edges (in graph 606). These increases can be offset byoperating in small road networks, emphasizing flexibility andcorrectness.

The original edges and transition edges have differences in terms ofattribution. Table 2 illustrates some of the differences in attributesbetween original edges and transitional edges in accordance with someembodiments:

TABLE 2 Feature/Attribute Original Edge Transition Edge Geometry Yes NoStreet Name Yes No Turn Cost No Yes Highway Class Yes No Time Cost YesYes (Optional) Distance Cost Yes No AV-Human Transition No Yes

Hard-coded turn restrictions in the turn graph representation can beeasily handled as well. If a turn between two edges is restricted, thetransition edge between the two is excluded altogether.

Traversing the Graph (Selecting a Route)

During traversal, a cost model may be used to evaluate the costs of theedges that are explored (e.g., weights are assigned as the costs of theedges based on tags or attributes associated with the edges). This isdone using a cost function (e.g., determined at run time by the client'srequest). An optimal path may be returned that minimizes this costfunction. Examples of costs that the cost function accounts for include,but are not limited to:

-   -   Time: The amount of time to traverse an edge    -   Distance: The distance (e.g., in meters) of an edge    -   Turn: A cost associated with turns (this cost returns 0 for all        non-transition edges)    -   Human-Intervention: A cost associated with going from a        non-human intervention to a human intervention zone (returns 0        for all non-transition edges).

Similarly, the routing can completely restrict certain edges ortransitions if necessary. A client can customize their own linearcombination of costs (or corresponding cost functions) to create ahybrid weight for an edge like so:

{  “weights”: {   “time”: 1,   “distance”: 0.25,   “turn”: 1,  “human_intervention_transition”: 10  },  “restrictions”:[“no_left_turns”] }

Which represents the following weight for an edge e:

weight(e)=time(e)+0.25*distance(e)+turn(e)+10*human(e)

There are no conversions for dimensions included here. Each costfunction is individually unit-less, and can therefore be compared easilywith other costs. FIG. 7 illustrates an example of an optimal route thatpenalizes human-intervention-zone transitions very heavily. For example,a first route from origin 702 to destination 704 that involves astraight line (e.g., the shortest path) between origin 702 and 704includes a human-intervention zone 700. When the penalty of thehuman-intervention zone is large, the optimal route between origin 702and 704 instead takes a longer path in order to avoid thehuman-intervention zone 700.

As more data is gathered, optimal weights can be captured for differentmodes of autonomous vehicle operation (e.g., a “safety” mode or an“avoid human interventions” mode). Different routing modes thus may usedifferent cost models and functions, resulting in different routes.Routing therefore may be preceding by selection of a mode in accordancewith some embodiments.

In some embodiments, on top of the static data, a real-time data layerperiodically updates the edge attributes on the graph and can affect thepath calculation. For example, a real-time traffic data pipeline mayprovide real-time speeds for a subset of the edges, or autonomousvehicle sensor data may provide real-time road or lane closureinformation. If desired, the client can override the real-timeinformation with historical cost functions.

The shortest path problem is the problem of finding a path from anorigin to a destination, which optimizes (e.g., minimizes) the sum ofthe weights of the path edges according to a cost function. To computethis optimal path, routing relies on 3 components: a graph, its weights,and algorithms to traverse the graph. The graph is a planar topologythat represents the traversable space. For human routing, this graph isdirectly correlated to the road network, with edges and nodesrepresenting road segments and intersections, respectively. For example,FIG. 8 illustrates a conventional example of a routing graph where roadsare edges and intersections are nodes. In this section, the discussionis restricted to graphs whose edge weights are non-negative.

Other graph representations are possible. For example, an edge-basedgraph is represented by taking the dual of the graph, where nodes map toroad segments and edges map to transitions between subsequent roadsegments. This representation can be useful as it allows for theintroduction or removal of transition edges in order to capture turncosts and restrictions respectively. The graph edges represent theentire road (ignoring lanes). Even with just a road representation, thegraph is very large. For example, the planet-wide OSM build currentlycontains on the order of 4 billion unique nodes. To ameliorate dataexplosion, other graph representations build multiple levels of details,partition into sub-graphs, or compute additional metadata such asdistance to landmarks.

Finding tighter lower bounds makes A* even more selective and hencefaster. The ALT algorithm (A*, Landmarks, Triangle inequality)pre-computes shortest path costs from each node in the graph to a set ofnodes called landmarks and exploits the triangle inequality toprecompute even tighter lower bounds for A*. Specifically, for adestination node t, a landmark node L, and a node x:

d(x, t)+d(t, L)≥d(x, L)⇒d(x, t)≥d(x, L)−d(t, L)=hL(x)

This means the subtracted distance (d(x, L)−d(t, L)) can be used as alower bound for the actual distance d(x,t). In practice, many landmarksare chosen, and the maximum of the subtracted distances can be taken inorder to obtain the tightest lower-bound heuristic.

In order to find a shortest path, each graph edge must have anassociated weight, which represents the cost of traversal. In simplerouting algorithms, the cost of a routing edge is the time it takes avehicle to travel over it. Usually the cost is a scalar value, but itcan also have multiple costs (e.g., time and distance) and/or confidenceprobabilities. Pareto/multi-value optimization has been used in theformer and stochastic routing in the latter. For time-dependent routingproblems the costs themselves can change over time and require adifferent approach to solve optimally.

The first algorithms created to find shortest path on non-negativelyweighted graphs were Dijkstra's and the Bellman-Ford algorithm.Unfortunately, these approaches do not scale well with even city-sizedroad graphs. Recent advances have focused on applying these fundamentalgraph exploration algorithms in clever ways to scale to real-world graphsizes. One improvement is to apply Dijkstra's from both the origin andthe destination node and employ clever termination conditions to thefrontiers of both searches to reduce the size of the search space.

The A* search algorithm is another popular technique that guidesDijkstra's search using a heuristic cost which acts as a lower bound forthe optimal cost. Conventionally, A* uses a priority queue for the nodesto visit where the priority of a node x is the sum of its current costd(s, x) and its heuristic cost h(x). In turn, this helps deprioritizenodes with prohibitively high costs. A typical lower bound for theshortest time cost is the haversine distance from a node to thedestination divided by a maximum car speed. One can think of A* as ageneralization of Dijkstra's naive routing algorithm with a zero-costheuristic (i.e., h(x)=0 for all x).

A* can also be used to optimally solve a weighted cost functionsdetermined at run-time given that the weights are non-negative.Specifically, for a set of cost functions f_(i):

f(x)=Σw _(i) f _(i)(x)

The minimum over f(x) is an upper bound for the weighted sum of theminimums of f_(i)(x). By defining the values f*, f_(i)*, as the minimumvalues taken for f(x), f_(i)(x), respectively, and x *, x_(i)*, as theminimizing arguments for those functions, respectively:

f _(i) *≤f _(i)(x)

f _(i) *≤f _(i)(x*)

w _(i) f _(i) *≤w _(i) f _(i)(x*)

Σw _(i) f _(i) *≤w _(i) f _(i)(x*)=f(x*)

Thus, the weighted sum of the minima can be used for each function as avalid lower bound for any hybrid cost function required at run time.This makes ALT a compelling algorithm for flexible costing as well.

Real-Time Data for Powering Fleets of Autonomous Vehicles

As autonomous vehicles (AVs) begin to operate, they heavily rely onpreviously collected and processed high-definition (HD) map data (e.g.,data that includes lane and direction information) for functionalitylike localization, which allows them to track their positions in thereal world. To operate more safely and efficiently, however, autonomousvehicles benefit from a separate layer of real-time data. For example,location data (e.g., GPS data or data from another positioning system,such as GLONASS, Galileo, or BeiDou) collected from autonomous vehiclesand non-autonomous vehicles in real-time indicates real-time traffic,which helps produce more accurate estimated arrival times and moreefficient routes.

In some embodiments, camera systems are used on autonomous vehicles(e.g., autonomous-vehicle camera sensors) and human-driven vehicles(e.g., dashcams) to collect real-time data on road conditions (e.g.,road closures, obstacles, traffic, etc.). This information, when used toroute autonomous vehicles, allows autonomous vehicles to avoid certainroads or navigate safely around them. Table 3 is a partial list of suchroad conditions (CV=computer vision):

TABLE 3 Feature (historical, real-time) Acquisition Method Weather(rain, fog, sunlight angle) CV and license Posted speed limit CV, GPS,license Road closures and other signs CV and license Road attributes CV,license, accelerometers, GPS potholes road quality lane line markerslane width barrier curvature unprotected left complex intersection(traffic light relevance) Mobility density (bicycle, vehicle, CVpedestrian, etc.) used for events like parades, baseball games, schoolcrossings, etc. Police, fire, and other emergency CV personnel Roadconstruction CV and license Pick-up/Drop-off (automatically CV andlicense finding good ones from video) Blackout areas Fleet Car accidentsHuman, GPS/accelerometer, license.

As referred to herein, “license” refers to data available from anothersource. For example, data available from a government (e.g. amunicipality, a local government, a state government, the federalgovernment) about construction zones may be licensed from thegovernment. The licensed data may be complete or incomplete. Whenincomplete, other methods of acquiring the data (e.g., computer vision(CV) with GPS or other location technology) can be used to complementthe data.

As illustrated in Table 3, computer vision detects many of the featuresin camera data that impact an autonomous vehicle's safety andefficiency. In some embodiments, labeled and unlabeled camera data,convolutional neural nets (CNNs), and/or both cloud and edge computingare leveraged to solve computer vision detection problems. Table 4enumerates further examples (RT is real-time):

TABLE 4 Sensors Real-time Data Impact GPS RT traffic More accurate ETA,more efficient routes GPS + cameras Lane-level traffic data Moreaccurate ETA, more efficient routes GPS + cameras Traffic incidentsPhotos of traffic to give rides context about upcoming traffic, data onincidents that may confuse a SDV Cameras Road closure Fewer reroutes,less human intervention, better routes Cameras Available parking spotsPlace for AVs to park when not on trips.

The list below, as well as Table 5, provides still more examples of roadconditions that can be detected using computer vision and accounted forin cost models:

-   -   Weather or road conditions due to weather (e.g., sun, rain,        wind, snow, fog);    -   Sunlight angles (e.g., an angle of less than, or less than or        equal to, a specified number of degrees (e.g., 20 degrees) to        the horizon may impair visual sensors, which may affect a        vision-based system);    -   Complex environments (e.g., dense pedestrian or bicycle        traffic);    -   Pavement conditions or type (e.g., new, modern, ancient,        cracked, well-paved, pot-holed, dirt, asphalt, cobblestone); and    -   Incidents (e.g., construction zones, police, road closures,        accidents, events such as parades, temporary regulatory signs).

Some embodiments detect only features that impact autonomous vehiclesafety and efficiency (e.g., forgo detection of features that do notimpact autonomous vehicle safety and efficiency). This constraineddetection space enables the capture of many specific images to solvespecific detection problems. For example, in some embodiments, thecomputer vision does not recognize objects that are not relevant to thedetection problems. Additionally, supervised learning techniques (likethe aforementioned CNN) are preferred over unsupervised learning, eventhough the latter data exists in much larger quantities and could pavethe future towards stronger artificial intelligence.

Another data advantage in these detection and classification problems isaccess to historical images as well as the real-time ones. For example,a historical image's location (e.g., GPS location) is snapped to theroad network so to correlate images of the same road at multiple times.This domain-specific data helps us more reliably detect outliers (likeweather and road construction) in the real-time data and allows us touse some unsupervised learning on unlabeled data (e.g., clustering forweather classification, below).

Some embodiments take advantage of both cloud and edge computing. Intraditional cloud computing, all data (e.g., from internet-of-thingsdevices) are transmitted to the cloud for batch processing (e.g.,distributed machine learning for computer vision). Unfortunately, thisnot only requires large and complex clusters to meet the computationalneeds, but also becomes a bottleneck for using data in real-time. Thevalue of the data diminishes quickly over time. With edge computing, thecomputing power on the mobile device is used to perform similar but lessaccurate detections to filter out noise early in the detection pipeline.In the cloud, potentially positive detections are refined with morecomplex models trained by more data and running on more powerfulcomputers. Simple interest points like Harris corners may also becaptured and used later for motion detection and tracking.

The approach in the edge versus the cloud may be largely the same, savethe complexity of the edge model and its relatively lower precision butpotentially higher recall.

TABLE 5 Feature Acquisition Method Detection Approach Weather (rain,fog, snow, CV and license Histograph of ice, sunlight angle)gradients/colors; CNN Posted speed limit CV, GPS, license CNN, sensorfusion, SVM for speed classification Road closures and other CV andlicense CNN signs Road attributes CV, license, CNN, sensor fusion,potholes accelerometers, polynomial fitting road quality GPS lane linemarkers lane width barrier curvature unprotected left complexintersection (traffic light relevance) Mobility density (bicycle, CV CNNvehicle, pedestrian, etc.) used for events like parades, baseball games,school crossings, etc. Police, fire, and other CV CNN emergencypersonnel Road construction CV and license CNN, scene recognitionPick-up/Drop-off CV and license N/A (automatically finding good onesfrom video) Blackout areas Fleet N/A Car Accidents Human, GPS, N/Aaccelerometers, license

Weather Recognition

In some embodiments, weather conditions that impair the autonomousvehicle's vision system are detected (e.g., rain, fog, or sunlightglare). The intuition is that weather conditions are global across theimage. A general approach is to extract global descriptors across theimage, compile them into a feature vector and learn the discriminatingones for multi-class classification with training data. In someembodiments, local weather feeds (“rainy today,” “sunny,” “fog in themorning in San Francisco”) are used as another descriptor in the imagefeature vector (e.g., are appended to the image feature vector).

For the global image descriptors, histogram of gradients (HOG) and colorhistograms may be used, as shown in FIG. 9 .

Given these feature vectors in a training set, a linear multi-classifier(Multi-class SVM) may be trained. There are more rich Machine Learningmodels that can be employed, such as those that perform smarter featureselection (e.g., M. Varma and B. R. Babu, “More generality in efficientmultiple kernel learning,” ACM International Conference on MachineLearning, 2009). CNNs may be used (see, e.g., Elhoseiny et al., “Weatherclassification with deep convolutional neural networks,” InternationalConference on Image Processing, 2015).

In some embodiments, using historical images of the same road at thesame hour, but over different days, histograms are clustered in ahistogram feature space using DBSCAN (Ester et al., “A density-basedalgorithm for discovering clusters in large spatial databases withnoise,” Proceedings of the Second International Conference on KnowledgeDiscovery and Data Mining, 1996) to detect outliers.

Moving to more spatially granular classification, posted speed limitsigns may be detected. This specific classification class of the moregeneral class of traffic signs can leverage another signal: location(e.g., GPS location). For example, a combination of a vision-basedclassifier and an Extended Kalman Filter (EKF) for speed allows forreliable detection and recognition of speed signs and their posted speedlimits.

Vision-based Classifier

Some embodiments use two approaches for classification: (1) boostedcascaded Haar-like feature classification (FIGS. 10 ) and (2) CNNs.Boosted cascaded Haar-like feature classification may be used initiallyto bootstrap image acquisition since the number of parameters to solveis less than a CNN. This approach has been used successfully for facedetection (Viola, Jones, “Robust Real-Time Face Detection,”International Journal of Computer Vision, 2004) in the past. The basicidea is that a set of weak classifiers (Haar features) which are onlyrequired to be slightly better than random are boosted together (viaAdaBoost, for example) to provide a higher prediction quality. Thecascaded aspect means that features are chained together inclassification and early bad classifiers can be used to rejectsubsequent features.

The cascaded weak classifiers are run over a sliding window (atdifferent levels of a Gaussian pyramid of the image) and the windowswith highest detection rates are kept. Finally, one window is selectedvia non-maximal suppression. The classification and detection (e.g.,localization of the class within the image) is done in two separatesteps.

FIG. 11 illustrates a boosted cascaded local binary patterns (LBP)classifier.

Extended Kalman Filter for Estimating Position and Speed to ImproveDetection

Before describing the general CNN vision approach, the use of SensorFusion to help detect speed limit signs is briefly explained. Thestandard Extended Kalman Filter (EKF) is used to fuse the inertialmeasurement unit (IMU) (accelerometer, gyroscope, and magnetometer) withlocation (e.g., GPS location) to predict location and velocity. For thesensor co-variances, map domain knowledge is used to provide betterestimates over standard calibration (e.g., in urban canyons there is anincrease in noise, but in some embodiments, normally distributed noiseis still assumed). Intuitively, drivers change their speed when theposted speed limit changes. The better location and speed estimates overadjacent road segments (over many trips) can be fed as additionalparameters for the CNN to specifically detect posted speed limit signsat speed changes.

General Traffic Signs and Objects

With boosted cascaded classifiers, the number of parameters is smallerthan a CNN, but it requires the use of Haar filters and the limitationsthat come them. With CNNs, copious amounts of labeled data are used fortraining to let the net optimize for the best convolutions to perform.

CNNs are a special type of deep neural net specifically used forcomputer vision classification. Similar to their general counterparts,CNNs consist of multiple layers of nodes, where each node is describedby a simple weight and bias (whose value is learned). CNNs also employReLUs for capturing non-linearities and employ the same techniques ofregularization and stochastic optimization (e.g., SGD) to minimize loss.CNNs differ from general neural nets in that some nodes actuallydescribe an image convolution or filter response on the previous layer.Additionally, researchers have employed image-domain specific insights,like max pooling, to increase CNN recognition accuracy. This and otherinsights allow machine learning engineers to intuitively understand howthe CNN progressively searches for image features from simple featuresto complex aggregated patterns. Recent work has shown high accuracy(91%) for traffic sign detection (Zhu et al, CVPR, 2016). Thus, CNNs aretrained to detect general traffic signs and any other object (but notscene), e.g., using a You Only Look Once (YOLO) architecture (e.g.,implemented with Keras using Tensorflow), shown in FIG. 12 .

FIG. 13 illustrates detection of various traffic signs, in accordancewith some embodiments.

Platform

Some embodiments of the present disclosure are implemented using Java,and more specifically GraphHopper. For an offline implementation, a JavaVirtual Machine (JVM) may be run onboard the autonomous vehicle. Tolessen the amount of memory needed on an onboard device, someembodiments shard the routing graph to load only the necessary graphcomponents into memory.

Alternatively, some embodiments run autonomous vehicle routing on aseparate mobile device, which interfaces with the car via localcommunication (Bluetooth, LAN, USB, etc.). In this case, runningautonomous vehicle routing in an offline mode is equivalent to runningit on the mobile device.

Operating Environment

FIG. 14 is a block diagram of a client-server environment 1400 inaccordance with some embodiments. The client-server environment 1400includes vehicles 1410 (e.g., 1410-1, 1410-2, . . . , 1410-n) that areconnected to a vehicle routing server 1420 through one or more networks1414. In some embodiments, vehicles 1410 are or are analogous tovehicles 306/308 (FIG. 3B) In some circumstances, the vehicles 1410operate as clients of vehicle routing server 1420. The one or morenetworks 1414 can be any network (or combination of networks) such asthe Internet, other Wide Area Networks, Local Area Networks,metropolitan area networks, VPNs, peer-to-peer, ad-hoc connections, andso on.

The non-autonomous vehicle 1410-1 is a representative human-drivenvehicle (e.g., a car, truck, or motorcycle). In some embodiments,non-autonomous vehicle 1410-1 is or is analogous to non-autonomousvehicle 306 (FIG. 3B) In some embodiments, non-autonomous vehicle 1410includes a dashcam 1412 that acquires and sends camera images to vehiclerouting server 1420, which can automatically identify road conditionsfrom the dashcam images (as well as from autonomous vehicle camerasensor data from autonomous vehicles, such as from sensors 1402 inautonomous vehicle 1410-2).

The autonomous vehicle 1410-2 (e.g., a car, truck, or motorcycle)includes non-transitory memory 1404 (e.g., non-volatile memory) thatstores instructions for one or more client routing applications 1406. Insome embodiments, autonomous vehicle 1410-2 is or is analogous toautonomous vehicle 308 (FIG. 3B) Client routing applications 1406 areexecuted by one or more processors (e.g., CPUs) 1408 on the autonomousvehicle 1410-2. In some embodiments, the client routing applications1406 send requests for routes to the vehicle routing server 1420 andreceive selected routes from the vehicle routing server 1420. In someembodiments, the client routing applications 1406 direct the autonomousvehicles 1410-2 along the selected routes. Client routing applications1406 may be embodied as any appropriate combination of programs,firmware, operating systems, or other logical or physical aspects of theautonomous vehicle 1410-2. Autonomous vehicle 1410-2 also optionallyincludes one or more network interfaces and one or more communicationbuses for interconnecting these components. Autonomous vehicle 1410-2further includes vehicle components, such as an engine/motor, a steeringmechanism, lights, signaling mechanisms, etc., some or all of which maybe under the control of programs (e.g., a client routing application1406) stored in memory 1404.

In some circumstances, a fleet of vehicles e.g., up to vehicle 1410-n)is in communication with vehicle routing server 1420. The fleet may be amix of autonomous and human driven vehicles or may be entirelyautonomous.

Vehicle routing server 1420 includes non-transitory memory 1416 (e.g.,non-volatile memory) that stores instructions for one or more serverrouting applications 1418 (sometimes referred to as “routing engines”).Vehicle routing server 1420 further includes one or more processors(e.g., CPUs) 1422 for executing server routing applications 1418. Theserver routing applications, stored in the non-transitory memory,include instructions for performing any of the methods described herein(e.g., method 1500, 1600, 1700, and/or 1800, FIGS. 15A-15B, 16-18 ).Server routing applications 1418 may be embodied as any appropriatecombination of programs, firmware, operating systems, or other logicalor physical aspects of the autonomous vehicle 1410-2. Vehicle routingserver 1420 also optionally includes one or more network interfaces andone or more communication buses for interconnecting these components.

The above identified applications correspond to sets of instructions forperforming functions described herein. The applications need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these instructions may be combined or otherwisere-arranged in various embodiments.

METHOD FOR VEHICLE ROUTING USING A COST MODEL WITH COSTS OTHER THANTRAVEL TIME

FIGS. 15A-15B are a flow diagram illustrating a method 1500 for routinga vehicle using a cost model with costs other than travel time, inaccordance with some embodiments. The method 1500 is performed at anelectronic device (e.g., vehicle routing server 1420, FIG. 14 ). Whilethe following discussion describes implementations where the vehiclerouting server 1420 performs the steps of the method 1500, in otherembodiments, one or more (e.g., all) of the steps are performed byanother electronic device, such as autonomous vehicle 1410-2 (e.g., acomputer system on vehicle 1410-2 with non-transitory memory storinginstructions for executing the method and one or more processors forexecuting the instructions). Moreover, some operations in method 1500are, optionally, combined and/or the order of some operations is,optionally, changed. In some embodiments, some or all of the operationsof method 1500 are performed automatically, without human intervention.

As described below, method 1500 provides technical solutions to problemsthat arise when routing autonomous vehicles. In particular, method 1500addresses shortcomings in the abilities of autonomous vehicles bysending the autonomous vehicles on routes that mitigate theseshortcomings (e.g., reduce the likelihood that an autonomous vehiclewill encounter a situation for which it is ill-equipped). Routingautonomous vehicles in a way that is tailored to their capabilitiesincreases passenger comfort and safety, conserves resources (e.g., cutsdown on carbon emissions and/or battery usage from autonomous vehicles),reduces traffic.

The vehicle routing server 1420 generates (1502) a cost model forrouting the autonomous vehicle. The cost model includes one or morecosts other than travel time (or other than both travel time anddistance). In some embodiments, the cost model includes a representationof a graph of nodes and edges. The respective edges have respective edgeweights that represent costs. In some embodiments, the graph is storedwithout edge weights, and the edges are associated with tags orattributes (e.g., road conditions). In some embodiments, generating thecost model includes assigning the edges weights (e.g., based on theirtags or attributes) at run-time (e.g., when a routing request isreceived). In some embodiments, the costs correspond to particulargeographical areas, roads, lanes within roads, maneuvers, and/or pathsthrough an intersection.

In some embodiments, at least one of the one or more costs other thantravel time is (1504) a cost for a particular geographical area, road,lane within a road, or maneuver. For example, various edges of the graphdescribed above correspond to a road, a line within a road, or amaneuver.

As an example of a cost associated with a particular area, in someembodiments, the cost model includes (1506) a cost of traversing (e.g.,entering) an area where autonomous driving is prohibited. In somecircumstances, autonomous vehicles are blacklisted from a particulararea. If this occurs in a driverless vehicle, the area must be avoidedaltogether and the cost assigned is prohibitively high. When the vehiclehas a human back-up driver, the cost may be set according to apreference for avoiding situations in which the human back-up drivermust take control (e.g., the preference can be set by the human back-updriver or another human, such as the car's owner). In some embodiments,the area where autonomous driving is prohibited is geo-fenced (e.g., hasa virtual geographic boundary defined, for example, by GPS, that enablessoftware to trigger a response when an autonomous vehicle enters orleaves a particular area, such as instructing a human to take overdriving when the autonomous vehicle has entered the geo-fenced area andoffering to resume control when the autonomous vehicle has left thegeo-fenced area). In some embodiments, the cost for the particulargeographical area is added to any costs associated with particular roadswithin the area (e.g., a road within a geo-fenced area may have a costassociated with pedestrian traffic which is added to the cost due to thefact that the road is within a restricted area).

In some embodiments, the cost model includes (1508) a cost of a drivingmaneuver (e.g., a lane change, a right turn, a left turn, an unprotectedleft turn, a U-turn, or a highway merge). In some embodiments, maneuversthat are unsafe for autonomous vehicles are assigned prohibitively highcosts. In some embodiments, the cost for a right turn is less than orequal to a cost for a protected left turn, which is less than a cost foran unprotected left turn. The effect of a higher cost is that vehiclerouting server 1420 is less likely to select a route that has the highercost feature. Thus, by making unprotected left turns higher costmaneuvers than right turns, the autonomous vehicle is less likely to berouted along a route that has unprotected left turns, resulting in fewerunprotected left turns. In some embodiments, the cost of an unprotectedleft turn is set to be prohibitively high, to avoid unprotected leftturns completely.

In some embodiments, the cost model includes (1510) a cost of (orassociated with) a characteristic of the autonomous vehicle. In someembodiments, the characteristic of the autonomous vehicle is (1512) avehicle type (e.g., a make or model of the vehicle, whether the vehicleis a sedan, sports utility vehicle, van, or larger vehicle). Forexample, in some embodiments, smaller roads have an associated lowercost for smaller vehicles as compared to higher vehicles. In someembodiments, the characteristic of the autonomous vehicle is (1514) anautonomous driving capability (or equivalently, an autonomous drivinglimitation) of the autonomous vehicle (e.g., a sensor capability ormaneuvering capability). For example, in some embodiments, roads withoutlane lines have a higher cost for autonomous vehicles that cannotdetermine lanes on their own. Routing according to costs ofcharacteristics of the autonomous vehicle is described in more detailwith reference to method 1600 (FIG. 16 ).

In some embodiments, the cost model includes (1516) a cost of a weathercondition, such as the current weather (e.g., it is snowing). In someembodiments, the cost of the weather condition is a cost of a roadcondition due to weather (e.g., there is snow on the road). In someembodiments, the one or more costs other than travel time include (1518)distinct road-specific costs for a shared weather condition at differentroads. For example, a small road may be assigned a higher cost in a snowstorm than a highway (especially if the small road is one that isinfrequently plowed, e.g., as determined from camera data as describedwith reference to method 1800, FIG. 18 ). In some embodiments, the oneor more costs other than travel time include (1520) distinctroad-specific costs for different weather conditions at different roads.For example, for a long trip, or a trip in a region with microclimates,it may be possible to avoid conditions such as snow and fog altogether.Roads having the weather condition are assigned one cost, while roadswithout the weather condition (e.g., roads with better weather) areassigned a lower cost.

In some embodiments, the one or more costs other than travel timeinclude a cost of a time of day (e.g., distinct road-specific costs ofthe time of day for different roads). For example, small country roadsmay be less safe at night than large highways. Thus, at night, smallcountry roads are assigned an extra cost that lowers the preference forsmall country roads over highways even more than the daytime preference.

In some embodiments, the one or more costs other than travel timeinclude a cost of a sunlight angle (e.g., distinct road-specific costsof the sunlight angle for different roads). For example, roads that arepointed nearly directly into the sun receive a higher cost at sunsetthan roads that are more angled with respect to the position of the sun.In some embodiments, the cost of the sunlight angle is assigned inaccordance with a time of year (e.g., the road only points directly intothe sun during sunsets in the winter, and thus the cost assigned to thesunset angle is assigned only during sunset times in the winter).

In some embodiments, the one or more costs other than travel timeinclude a cost of a pedestrian traffic pattern (e.g., a predicted ormeasured density of pedestrian traffic). In some embodiments, the one ormore costs other than travel time include a cost of a non-motorizedvehicle (e.g., bicycle) traffic pattern. In some embodiments, thepedestrian/non-motorized vehicle traffic pattern is a temporal pattern.In some embodiments, the pedestrian/non-motorized vehicle trafficpattern is a spatial pattern (e.g., the autonomous vehicle routingattempts to avoid areas of high pedestrian traffic, such as TimesSquare). In some embodiments, the pedestrian/non-motorized vehicletraffic pattern is based on historical data (e.g., thepedestrian/non-motorized vehicle traffic pattern is not ascertained bythe autonomous vehicle while driving along the route, but is ascertainedbefore routing so that the pedestrian/non-motorized vehicle trafficpattern can be used in the routing).

In some embodiments, the one or more costs other than travel timeinclude a cost of a pavement condition. In some embodiments, the one ormore costs other than travel time include a cost of a road age (e.g.,older roads are assigned a higher cost than newer roads).

In some embodiments, the one or more costs other than travel timeinclude a cost of a road lacking lane lines (e.g., roads with lane linesare assigned a lower cost than roads without lane lines).

In some embodiments, the one or more costs other than travel timeinclude a cost of an electric-charging constraint (e.g., roads withoutelectric-charging facilities for long stretches are assigned a highercost than roads with frequent electric-charging facilities). Thisconstraint may correspond to a range of an electric vehicle.

In some embodiments, the one or more costs other than travel timeinclude a cost of acceleration. Motion sickness is common in autonomousvehicles. This is at least in part because autonomous vehicles have notyet learned to mimic human driving, so autonomous vehicles tend toaccelerate (e.g., change speed and/or direction) in ways that feelunnatural to passengers. To mitigate motion sickness in autonomousvehicles, some embodiments of method 1500 assign a cost to acceleration(e.g., include the acceleration cost in the cost model) to disfavorroutes with unnatural acceleration profiles.

In some embodiments, the cost model further includes a travel-time cost(and/or distance cost), and the route is selected in accordance withboth the travel-time cost (and/or distance cost) and the one or moreother costs. The costs other than travel time (and/or distance) thus mayoperate in tandem with (e.g., in addition to) travel-time and/ordistance costs, preventing the vehicle routing server 1420 from routingthe autonomous vehicle along an unreasonably long route to, for example,avoid human intervention zones.

The vehicle routing server 1420 receives (1522) a request to route theautonomous vehicle from a first location to a second location. In someembodiments, the request to route the autonomous vehicle is receivedprior to generating the cost model. For example, the first location andthe second location are physical locations each corresponding to astreet address, an intersection, a landmark (e.g., “City Hall”), or thelike. In some embodiments, the request identifies the first location andthe second location. In some embodiments, the first location is anorigin of the trip. In some embodiments, the second location is thedestination of the trip. Alternatively, the autonomous vehicle routingcan be done piecemeal, with the first location and the second locationbeing intermediate points along a larger route. In some embodiments, therequest to route the autonomous vehicle is received from a vehicle(e.g., vehicle 1410-2 or 1410-n) or a passenger within the vehicle(e.g., using the passenger's smart phone).

In response to the request to route the autonomous vehicle, the vehiclerouting server 1420 selects (1524) a route from the first location tothe second location in accordance with the cost model. In someembodiments, selection of the route is performed as described aboveunder the heading “Traversing the Graph (Selecting a Route).”

In some embodiments, the one or more costs other than travel timeinclude a cost of an event present at the time of a routing request(e.g., a construction project, traffic accident, or sporting event). Insome embodiments, the one or more costs other than travel time include acost of an event beginning and/or ending at the time of the routingrequest (e.g., a sports game beginning and/or ending). Selecting theroute is performed based at least in part on the cost of the event. Forexample, the one or more costs other than travel time include (1526) thecost of a regularly-scheduled event. Selecting the route is performedbased at least in part on the cost of the regularly-scheduled event, inaccordance with a determination that the regularly-scheduled event isoccurring at the time of the routing request. For example, when asporting event is occurring when the request is received, the streetsaround the venue (e.g., stadium) are assigned an additional cost todisfavor routes that go near the venue.

In some embodiments, selecting the route is performed based at least inpart on the cost of the regularly-scheduled event, in accordance with adetermination that the regularly-scheduled event will be occurring whenthe autonomous vehicle is expected to reach the event. For example, aroute that will reach the area around the venue within a specified timeperiod corresponding to a start time of the event is disfavored.

Routing vehicles away from events present at the time of routingincreases passenger comfort and safety, conserves resources (e.g., cutsdown on carbon emissions and/or battery usage from autonomous vehicles),reduces traffic.

Further in response to the request, the vehicle routing server 1420routes (1528) the autonomous vehicle along the selected route. In someembodiments, routing the autonomous vehicle includes controlling themechanical operation of the vehicle, including controlling the steering,acceleration, breaking, and signaling of the autonomous vehicle. In someembodiments, routing the autonomous vehicle includes directing theautonomous vehicle to follow the selected route. In some embodiments,directing the autonomous vehicle to follow the selected route includesdirecting a separate computer-implemented module (e.g., routingapplication 1406 onboard vehicle 1410-2 or 1410-n) to follow theselected route (e.g., providing the onboard module with the selectedroute). The onboard module handles the mechanical control of the vehicle(e.g., handles the breaking, acceleration, steering, and signaling ofthe vehicle, avoiding pedestrians and the like). In some embodiments,the vehicle follows the route automatically and without humanintervention (e.g., passenger intervention).

In some embodiments, the cost model is generated in response to therequest to route the autonomous vehicle and the cost model is a firstcost model for the autonomous vehicle. In some embodiments, the vehiclerouting server 1420 receives a request to route a non-autonomous vehicle(e.g., a human-driven vehicle without autonomous capabilities) from athird location to a fourth location. The vehicle routing server 1420generates a second cost model for routing the non-autonomous vehicle.The second cost model is distinct from the first cost model and does notinclude one or more of the costs other than travel time that areincluded in the first cost model (e.g., does not include a cost oftraversing an area where autonomous driving is prohibited, does notinclude a cost of an acceleration). In some embodiments, the second costmodel includes different costs for factors that were taken into accountfor the first cost model. For example, the first cost model includes afirst cost of a respective driving maneuver and the second cost modelincludes a second cost of the same respective driving maneuver, wherethe second cost is different from the first cost. In response to therequest to route the non-autonomous vehicle, the vehicle routing server1420 selects a route from the third location to the fourth location inaccordance with the second cost model and routes the non-autonomousvehicle along the selected route for the non-autonomous vehicle (e.g.,the selected route for the autonomous vehicle is a first selected routeand the selected route for the non-autonomous vehicle is a secondselected route).

Routing autonomous vehicles and non-autonomous vehicles differently,e.g., by using different cost models for autonomous vehicles andnon-autonomous vehicles, respectively, tailors routes to the respectiveabilities of autonomous and non-autonomous vehicles. Routing vehicles ina way that is tailored to their capabilities increases passenger comfortand safety, conserves resources (e.g., cuts down on carbon emissionsand/or battery usage from autonomous vehicles), reduces traffic.

In some embodiments, the autonomous vehicle is a first autonomousvehicle having first autonomous driving capabilities and the cost modelis a first cost model for the first autonomous vehicle. The one or morecosts other than travel time for the first cost model include a costcorresponding to the first autonomous driving capabilities. In someembodiments, the vehicle routing server 1420 receives a request to routea second autonomous vehicle from a fifth location to a sixth location.The second autonomous vehicle has second autonomous driving capabilitiesare different from the first autonomous driving capabilities. Thevehicle routing server 1420 generates a third cost model for routing thesecond autonomous vehicle. The third cost model is distinct from thefirst cost model and includes a cost corresponding to the secondautonomous driving capabilities (e.g., that's different from the costcorresponding the first autonomous driving capabilities for the firstcost model for the first autonomous vehicle). In response to the requestto route the second autonomous vehicle, the vehicle routing server 1420selects a route from the fifth location to the sixth location inaccordance with the second cost model routes the second autonomousvehicle along the selected route for the second autonomous vehicle.

Routing autonomous vehicles differently depending on their capabilities,e.g., by using different cost models for different autonomous vehicleswith different capabilities, respectively, tailors routes to therespective abilities of the autonomous vehicles. Routing vehicles in away that is tailored to their capabilities increases passenger comfortand safety, conserves resources (e.g., cuts down on carbon emissionsand/or battery usage from autonomous vehicles), reduces traffic.

It should be understood that the particular order in which theoperations in FIGS. 15A-15B have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. For brevity, these details are not repeated here. In addition,the operations in FIGS. 15A-15B may be combined with other operationsdescribed elsewhere (e.g., with reference to FIGS. 16-18 ) or havecharacteristics described with reference to analogous operationsdescribed elsewhere (e.g., the routing operation of method 1500 mayoptionally share any of the characteristics of other routing operationsdescribed elsewhere herein).

Method of Autonomous Vehicle Routing Using a Cost Model with VehicleConstraints

FIG. 16 is a flow diagram illustrating a method 1600 of autonomousvehicle routing using a cost model with vehicle constraints, inaccordance with some embodiments. The method 1600 is performed at anelectronic device (e.g., vehicle routing server 1420, FIG. 14 ). Whilethe following discussion describes implementations where the vehiclerouting server 1420 performs the steps of the method 1600, in otherembodiments, one or more (e.g., all) of the steps are performed byanother electronic device, such as vehicle 1410-2 (e.g., a computersystem on vehicle 1410-2 with non-transitory memory storing instructionsfor executing the method and one or more processors for executing theinstructions). Moreover, some operations in method 1600 are, optionally,combined and/or the order of some operations is, optionally, changed. Insome embodiments, some or all of the operations of method 1600 areperformed automatically, without human intervention.

As described below, method 1600 provides technical solutions to problemsthat arise when routing autonomous vehicles. In particular, method 1600recognizes the fact that different autonomous vehicles have differentdriving capabilities. To that end, in some embodiments, method 1600routes an autonomous vehicle according the autonomous vehicle's specificcapabilities. Routing an autonomous vehicle in a way that is tailored toits specific capabilities increases passenger comfort and safety,conserves resources (e.g., cuts down on carbon emissions and/or batteryusage from autonomous vehicles), reduces traffic.

The vehicle routing server 1420 receives (1602) a request to route theautonomous vehicle from a first location to a second location (e.g., arequest to provide a route for a trip). In some embodiments, the vehiclerouting server 1420 receives a current location of the autonomousvehicle. For example, the request to route the autonomous vehicle fromthe first location to the second location includes the current locationof the autonomous vehicle (e.g., the first location). The request mayhave any of the characteristics of the request described with respect tooperation 1522, method 1500 (FIGS. 15A-15B).

In some embodiments, the vehicle routing server 1420 receives a currentlocation of the fleet vehicles. In some embodiments, the vehicle routingserver 1420 determines staging locations for the fleet of vehicles(e.g., locations from which the routes will begin).

In addition, in some embodiments, the vehicle routing server 1420receives (1604), with the request to route the autonomous vehicle, anidentifier for the autonomous vehicle. For example, the identifier ofthe autonomous vehicle may be a vehicle identification number (VINnumber); a make, model, and year of the autonomous vehicle; or anaccount number for the autonomous vehicle. In some embodiments, thevehicle routing server 1420 receives a current location of theautonomous vehicle. For example, the request to route the autonomousvehicle from the first location to the second location includes thecurrent location of the autonomous vehicle (e.g., the first location).

The vehicle routing server 1420 identifies (1606) a set of one or moreautonomous driving capabilities of the autonomous vehicle. In someembodiments, the vehicle routing server looks up (1608) one or moreautonomous driving capabilities of the autonomous vehicle based on theidentifier (e.g., in a vehicle capability database). In someembodiments, the set of one or more autonomous driving capabilities ofthe autonomous vehicle includes a capability of a sensor of theautonomous vehicle (e.g., a sensor 1402, FIG. 14 ). The vehicle routingserver 1420 uses the identifier that is received with the routingrequest to look up the vehicle in a database. The database returnsinformation indicating what types of sensors the vehicle has and/or whatthe sensors' capabilities are.

In some embodiments, identifying the set of one or more autonomousdriving capabilities of the autonomous vehicle includes (1610) receivingan autonomous driving capability with the request. For example, therequest includes information identifying what types of sensors thevehicle has. In some embodiments, one or more of the autonomous drivingcapabilities is included in the request, which also identifies thevehicle, and one or more additional driving capabilities are determinedfrom looking up the vehicle in a database. In some embodiments, the setof one or more autonomous driving capabilities includes a plurality ofautonomous driving capabilities.

In some embodiments, the set of one or more autonomous drivingcapabilities of the autonomous vehicle includes a maneuverability of theautonomous vehicle. For example, the set of autonomous drivingcapabilities may include a rating, score, or cost, for a particularvehicle or vehicle type, for an ability to make unprotected lefts,change lanes, or merge onto highways.

In some embodiments, the set of one or more autonomous drivingcapabilities of the autonomous vehicle includes a limitation on theautonomous vehicle determined from historical performance data for atleast one of the autonomous vehicle and vehicles of a same type as theautonomous vehicle. For example, sensor data from autonomous vehiclesthat were involved in accidents or near accidents can be used toidentify situations in which the vehicle or vehicle type performspoorly. As a more specific example, it may be determined based onhistorical performance data that a particular make, model, and year ofautonomous vehicle performs poorly in dense pedestrian traffic or withlow sunlight angles.

In some embodiments, the set of one or more autonomous drivingcapabilities of the autonomous vehicle includes a safety rating for theautonomous vehicle (e.g., a composite safety score based on NHTSA safetyratings).

In some embodiments, the vehicle routing server 1420 generates (1612) acost model for routing the autonomous vehicle that includes one or morecosts for the set of one or more autonomous driving capabilities of theautonomous vehicle (e.g., respective costs for respective capabilitiesin the set). In some embodiments, generating the cost model includesassigning costs in accordance with the set of one or more autonomousdriving capabilities. For example, the cost model may include a graphrepresenting a map (e.g., that is stored before the cost model isgenerated). A particular edge in the graph may correspond to a highwaymerge, and be tagged with “highway merge” as an attribute. The vehiclerouting server 1420 assigns, as part of the generating operation, a costto the edge based on the requesting vehicle's capabilities with respectto merging onto highways.

The vehicle routing server 1420 selects (1614) a route from the firstlocation to the second location in accordance with the set of one ormore autonomous driving capabilities of the autonomous vehicle (e.g.,the vehicle routing server 1420 selects (1616) the route from the firstlocation to the second location based at least in part on the costmodel). For example, when an autonomous vehicle does poorly with highwaymerges, the assigned costs for the highway merge edges are higher. As aresult, vehicle routing server 1420 is less likely to select a routethat includes a highway merge.

As another example, in some embodiments, the selecting is performedbased at least in part on the safety rating of the autonomous vehicle.In some embodiments, a vehicle with a low safety rating is routed alonga safer route than a vehicle with a higher safety rating (e.g., a saferroute is selected). In some embodiments, the set of one or moreautonomous driving capabilities includes situation-specific (e.g.,maneuver-specific, weather-specific, location-specific, etc.) safetyratings (e.g., a front passenger side safety rating is used to increaseor decrease the preference routes with unprotected lefts).

The vehicle routing server 1420 routes (1618) the autonomous vehicle inaccordance with the selected route. Routing operation 1618 is analogousto routing operation 1528, described above with reference to method 1500(FIGS. 15A-15B).

In some embodiments, the autonomous vehicle is a first autonomousvehicle. In some embodiments, the vehicle routing server 1420 receives arequest to route a second autonomous vehicle from a third location to afourth location. The vehicle routing server 1420 identifies a set of oneor more autonomous driving capabilities of the second autonomous vehicle(e.g., different sensor, different specifications on abilities to detectobjects/lanes, etc.). The set of one or more autonomous drivingcapabilities of the second autonomous vehicle include at least oneautonomous driving capability different from the set of one or moreautonomous driving capabilities of the first autonomous vehicle. Thevehicle routing server 1420 selects a route from the third location tothe fourth location in accordance with the set of one or more autonomousdriving capabilities of the second autonomous vehicle and routing thesecond autonomous vehicle in accordance with the selected route for thesecond autonomous vehicle.

Routing autonomous vehicles differently depending on their capabilities,e.g., by using different cost models for different autonomous vehicleswith different capabilities, respectively, tailors routes to therespective abilities of the autonomous vehicles. Routing vehicles in away that is tailored to their capabilities increases passenger comfortand safety, conserves resources (e.g., cuts down on carbon emissionsand/or battery usage from autonomous vehicles), reduces traffic.

It should be understood that the particular order in which theoperations in FIG. 16 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.For brevity, these details are not repeated here. In addition, theoperations in FIG. 16 may be combined with other operations describedelsewhere (e.g., with reference to FIGS. 15A-15B and 17-18 ) or havecharacteristics described with reference to analogous operationsdescribed elsewhere (e.g., the routing operation of method 1600 mayoptionally share any of the characteristics of other routing operationsdescribed elsewhere herein).

Method of Autonomous Vehicle Routing Using a Cost Model withIntersection Costs

FIG. 17 is a flow diagram illustrating a method 1700 ofautonomous-vehicle routing using a cost model with intersection costs,in accordance with some embodiments. The method 1700 is performed at anelectronic device (e.g., vehicle routing server 1420, FIG. 14 ). Whilethe following discussion describes implementations where the vehiclerouting server 1420 performs the steps of the method 1700, in otherembodiments, one or more of the steps are performed by anotherelectronic device, such as vehicle 1410-2 (e.g., a computer system onvehicle 1410-2 with non-transitory memory storing instructions forexecuting the method and one or more processors for executing theinstructions). Moreover, some operations in method 1700 are, optionally,combined and/or the order of some operations is, optionally, changed. Insome embodiments, some or all of the operations of method 1700 areperformed automatically, without human intervention.

As described below, method 1700 provides technical solutions to problemsthat arise when routing autonomous vehicles. In particular, method 1700addresses shortcomings in the abilities of autonomous vehicles byperform certain maneuvers at intersections by sending the autonomousvehicles on routes that reduce the likelihood that the autonomousvehicle will have to make such a maneuver. For example, it is generallyeasier to make a right-hand turn than an un-protected left turn. In somecircumstances, method 1700 will make un-protected left turns “moreexpensive” (in terms of a cost in a cost model), than right-hand turns,thus reducing the number of unprotected left turns made by theautonomous vehicle. Routing autonomous vehicles in a way that istailored to their capabilities increases passenger comfort and safety,conserves resources (e.g., cuts down on carbon emissions and/or batteryusage from autonomous vehicles), reduces traffic.

The vehicle routing server 1420 generates (1702) a cost model forrouting the autonomous vehicle from a first location to a secondlocation. The cost model includes, for an intersection, a plurality ofcosts for traversing distinct paths through the intersection. Respectivecosts correspond to respective paths through the intersection. In someembodiments, the cost model includes representations of intersections,as described with reference to the section above under the heading“Constructing the Routing Graph.” In some embodiments, the cost model isrepresented as a graph of nodes and edges. The respective edges haverespective edge weights that represent costs. The vehicle routing server1420 represents the intersection as a plurality of nodes and a pluralityof edges in the graph. Each edge of the representation of theintersection represents a distinct path through the intersection. Forexample, FIG. 4 illustrates a plurality of paths through an intersectionrepresented as edges (e.g., a path from the southern side (S) of thenorth-south road to the eastern side (E) of the east-west road; a pathfrom the southern side (S) of the north-south road to the northern side(N) of the north-south road).

Further, the generating operation 1702 may have any of thecharacteristics of the generating operations described elsewhere herein(e.g., edge weights are assigned in response to the request andoptionally include costs associated with a particular vehicle'sconstraints).

In some embodiments, the vehicle routing server 1420 forgoesrepresentation of a forbidden path through the intersection. Forexample, in FIG. 4 , no edge exists between the southern side (S) of thenorth-south road and the western side (W) of the east-west road becausethe east-west road is a one-way road going east).

In some embodiments, the intersection is an intersection of a pluralityof roads. Each road in the plurality of roads has one or more costsdistinct from the plurality of costs for traversing the distinct pathsthrough the intersection (e.g., the roads that intersect at theintersection have their own costs that are distinct from the costs forthe paths through the intersection). In some embodiments, the edgesrepresenting paths through the intersection have differentcharacteristics from the edges representing roads (e.g., as shown inTable 1 above).

In some embodiments, the plurality of costs for traversing the distinctpaths through the intersection include one or more costs of a drivingmaneuver for traversing one of the distinct paths through theintersection (e.g., a lane change, a right turn, a left turn, anunprotected left turn, a U-turn, and a highway merge). The cost ofdriving maneuvers, and the technical benefits thereof, are described ingreater detail with reference method 1500.

The vehicle routing server 1420 receives (1704) a request to route theautonomous vehicle from a first location to a second location. In someembodiments, the cost model is generated after receiving the request(e.g., in response to receiving the request). In some embodiments, thevehicle routing server 1420 receives a current location of theautonomous vehicle. For example, the request to route the autonomousvehicle from the first location to the second location includes thecurrent location of the autonomous vehicle (e.g., the first location).The request may have any of the characteristics of the request describedwith respect to the request receiving operations in methods 1500 and1600 (FIGS. 15A-15B, 16 ).

In response to the request, the vehicle routing server 1420 selects(1706) a route from the first location to the second location inaccordance with the cost model, based at least in part on one of theplurality of costs for traversing the distinct paths through theintersection (e.g., as described above with respect to the selectingoperations in methods 1500 and 1600).

Further in response to the request, the vehicle routing server 1420routes (1708) the autonomous vehicle in accordance with the selectedroute (e.g., as described above with respect to the routing operationsin methods 1500 and 1600).

It should be understood that the particular order in which theoperations in FIG. 17 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.For brevity, these details are not repeated here. In addition, theoperations in FIG. 17 may be combined with other operations describedelsewhere (e.g., with reference to FIGS. 15A-15B, 16, and 18 ) or havecharacteristics described with reference to analogous operationsdescribed elsewhere (e.g., the routing operation of method 1700 mayoptionally share any of the characteristics of other routing operationsdescribed elsewhere herein).

Method of Overlaying Real-Time Data Onto an Autonomous Vehicle RoutingCost Model

FIG. 18 is a flow diagram illustrating a method 1800 of overlayingreal-time data onto an autonomous-vehicle-routing cost model, inaccordance with some embodiments. The method 1800 is performed at anelectronic device (e.g., vehicle routing server 1420, as shown in FIG.14 ). While the following discussion describes implementations where thevehicle routing server 1420 performs the steps of the method 1800, inother embodiments, one or more of the steps are performed by anotherelectronic device, such as a vehicle 1410-2 (e.g., a computer system onvehicle 1410-2 with non-transitory memory storing instructions forexecuting the method and one or more processors for executing theinstructions). Moreover, some operations in method 1800 are, optionally,combined and/or the order of some operations is, optionally, changed. Insome embodiments, some or all of the operations of method 1800 areperformed automatically, without human intervention.

As described below, method 1802 provides technical solutions to problemsthat arise when routing autonomous vehicles by recognizing theimportance of vehicles in the Internet of Things (IoT). In particular,method 1802 recognizes that vehicles, especially autonomous vehicles,are not just modes of transportation but also represent a vast array ofmoving sensors, capable of detecting conditions that are relevant to thetechnical problem of routing autonomous vehicles. Thus, in someembodiments, method 1802 routes an autonomous vehicle using roadconditions automatically determined using information from sensors onother vehicles, allowing the autonomous vehicle to be routed moreefficiently. Routing autonomous vehicles more efficiently conservesresources (e.g., cuts down on carbon emissions and/or battery usage fromautonomous vehicles), increases passenger comfort and safety, reducestraffic.

The vehicle routing server 1420 receives (1802) information obtainedfrom a camera (e.g., a dashcam or an autonomous vehicle camera sensor)on a second vehicle (e.g., additional vehicle) distinct from theautonomous vehicle to be routed. In some embodiments, information isobtained from cameras in a fleet of cameras, some human driven (e.g.,with dashcams) and/or some autonomous (e.g., with dashcams and/orautonomous vehicle camera sensors). In some embodiments, the informationis obtained in real-time as it is acquired by the camera (e.g., receivedwithout an intentionally introduced delay so that it is available tovehicle routing server 1420 within a matter of seconds after beingacquired by the camera). In some embodiments, vehicle routing server1420 continuously receives the information obtained from the camera(e.g., the vehicle routing server 1420 continuously monitors for newcamera information). In some embodiments, the information obtained fromthe camera includes one or more images that are analyzed by vehiclerouting server 1420 (e.g., as described below with respect to operation1804). In some embodiments, the images include still pictures. In someembodiments, the images comprise a video feed. In some embodiments, theimages are analyzed by a computer on the vehicle 1410 and theinformation obtained from the camera includes the automaticallyidentified road conditions (e.g., as described below with reference tooperation 1804).

The vehicle routing server 1420 automatically identifies (1804) a roadcondition using image analysis of the information received from thecamera on the second vehicle. Identification of road conditions isdescribed above with reference to the sections under the headings“Real-Time Data for Powering Fleets of Autonomous Vehicles,” “WeatherRecognition,” “Vision-based Classifier,” and “Extended Kalman Filter forEstimating Position and Speed to Improve Detection.”

The vehicle routing server 1420 receives (1806) a request to route theautonomous vehicle from a first location to a second location. In someembodiments, the vehicle routing server 1420 receives a current locationof the autonomous vehicle. For example, the request to route theautonomous vehicle from the first location to the second locationincludes the current location of the autonomous vehicle (e.g., the firstlocation). Route request receiving operation 1806 has any of thecharacteristics of the route request receiving operations above (e.g.,with reference to methods 1500, 1600, and 1700, FIGS. 15A-15B, 16-17 ).

The vehicle routing server 1420 generates (1808) a cost model forrouting the autonomous vehicle (e.g., generates a cost model asdescribed above, by assigning weights to edges of a graph-based onattributes). The cost model includes a cost of the road conditionautomatically identified from the information received from the cameraon the second vehicle. In some embodiments, the cost model is generatedin response to the request. In some embodiments, generating the costmodel includes updating a previous cost model in response to therequest. In some embodiments, the cost model is updated in real-time asroad conditions are identified from cameras on other vehicles. In someembodiments, generating the cost model includes assigning an attributeto a road, lane in a road, driving maneuver, or geographical area basedon a road condition automatically identified for that road, lane,driving maneuver, or geographical area using camera information.

In some embodiments, the vehicle routing server 1420 determines whetherthe road condition is present at the time of the request. The cost ofthe road condition is included in the cost model in accordance with adetermination that the road condition is present at the time of therequest.

For example, the vehicle routing server 1420 may determine that it issnowing on Mountain Road based on dashcam data from a vehicle beingdriven on Mountain Road. The vehicle routing server 1420 then assignsthe attribute “snowing” to Mountain Road until it can be determined thatit is no longer snowing on Mountain Road. When a route request isreceived while Mountain Road has the “snowing” attribute, the vehiclerouting server 1420 assigns a cost to Mountain Road for the snowingattribute (e.g., a fixed cost or a cost based on the requestingautonomous vehicle's ability to drive in the snow).

In some embodiments, the vehicle routing server 1420 determines fromsecond camera information (e.g., from a third vehicle distinct from thesecond vehicle (e.g., additional) and the autonomous vehicle) that theroad condition is no longer present. For example, the vehicle routingserver 1420 determines from another vehicle that it is now “sunny” onMountain Road and that Mountain Road is clear of snow, and removes theattribute for “snowing.”

In some embodiments, the road condition automatically identified fromthe information received from the camera on the second vehicle (e.g.,additional) is a condition selected from the group consisting of aweather condition, a road condition due to weather, a road condition dueto construction, and a road condition due to a traffic accident.

In some embodiments, the road condition automatically identified fromthe information received from the camera on the second vehicle (e.g.,additional) is a condition selected from the group consisting of asunlight angle, pedestrian traffic, non-motorized vehicle traffic, apavement condition, and a lack of lane lines.

Further in response to the request, the vehicle routing server 1420selects (1810) a route from the first location to the second location inaccordance with the cost model (e.g., as described with respect to theselecting operations above).

Still further in response to the request, the vehicle routing server1420 routes (1812) the autonomous vehicle in accordance with theselected route (e.g., as described with the routing operations above)

It should be understood that the particular order in which theoperations in FIG. 18 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.For brevity, these details are not repeated here. In addition, theoperations in FIG. 18 may be combined with other operations describedelsewhere (e.g., with reference to FIGS. 15A-15B and 16-17 ) or havecharacteristics described with reference to analogous operationsdescribed elsewhere (e.g., the routing operation of method 1800 mayoptionally share any of the characteristics of other routing operationsdescribed elsewhere in this document).

Many-to-Many Vehicle Routing and Management

There have been efforts in the past to solve problems such as theVehicle Routing Problem (VRP), and some of its variants (e.g.,Dial-A-Ride, Capacitated VRP, etc.). Some implementations describedbelow solve sizable problems at scale. Broadly speaking, someimplementations involve breaking the larger problem down into smallersub-problems using spatio-temporal clustering, and then parallelizingthese sub-problems across specialized solver instances to find solutionsquickly.

Fleet Routing Problem (FRP)

A fleet routing problem (FRP) can be defined as follows: given set ofvehicles with heterogeneous capacity (e.g., a non-constant fleet size),with vehicles at various locations and multiple pickup/drop-off requeststhat can be time-window constrained (e.g., no one should wait more than10 minutes for a pick-up), assign a subset of the vehicles with the taskof picking up and dropping off everyone while minimizing an objectivefunction. An objective function can take various forms, including aso-called canonical form:

C=Σ _(V)(w ₁ R _(v) +w ₂ D(R _(v))).

In the equation above, C is an overall cost function for the entirefleet, V is a set of vehicles, R_(V) is a cost of a route taken by avehicle v, and D is a discomfort cost function of the passengersaccording to the route. For example, the cost function D balances apassenger's waiting time and overall travel time using weightcoefficients w₁ and w₂.

Because VRP is an NP-Hard optimization problem, generalized VRP solvers(e.g. MILPs or CPs) take an unacceptably long time to solve, even whenapplied to problems of modest size (e.g. 10 vehicles for 100 requests).For situations involving real-time request patterns and updates, thereis a need to solve much larger problems in real-time.

Architecture

An example architecture for solving an approximated version of the fleetrouting problem is shown in FIG. 19 . The router 1902 determines routesas described elsewhere in this document, calculates estimated times ofarrival and costs for many-to-many requests (e.g., a plurality ofpassengers making requests, each of which can be serviced by any one ofa plurality of vehicles). The solver 1910 services the instances of theFRP problem from the planner 1908, as described below. In someembodiments, each vehicle of the plurality of vehicles (e.g., in afleet) is solved in parallel by planner 1908.

The grouper 1906 breaks larger FRP problem into a plurality of smallerFRP problem instances via spatio-temporal clustering.

In some embodiments, the grouper 1906 assigns passengers to a set ofpotential stops and performs agglomerative clustering to partition stopsinto independently solvable groups by planner 1908, allowing aparallelizable divide-and-conquer.

In some embodiments, passenger pickup/drop-offs are assigned to a set ofN potential stops, ranked by walking distance. FIG. 20 illustratesrequested pick-ups and drop-offs assigned to (e.g., represented by aline connecting the stop and the location of the request) potentialstops based on a distance between the requested pick-up/drop-offlocations.

FIG. 21 illustrates that, in some embodiments, requests are groupedspatially via agglomerative clustering with complete/maximum linkageusing real-time driving ETAs (e.g., provided by the router) as adistance/cost function. This clustering corresponds to minimizing themaximum distance between any two stops in a cluster. A target k clustersis used such that the maximum number elements in a cluster is solvableby planner to hit a target latency (e.g., 5 seconds).

After clustering, a plurality of vehicles is assigned to pairs ofclusters. In some embodiments, vehicles are filled to capacity whileminimizing their travel distance.

In some embodiments, cluster pairs are represented as centroids withedges corresponding to load/demand to traverse between clusters, asshown in FIG. 22 .

Each vehicle of the plurality of vehicles, now assigned a set of clusterpairs, becomes a sub-plan that is solved in parallel by planner 1908.The number of cluster pairs assigned to a vehicle is a function of thevehicle's capacity and load on the cluster pair.

Each vehicle sub-plan is then unioned to form the global fleet plan, asshown in FIG. 23 . For example, FIG. 23 illustrates the combined (e.g.,global fleet) plan of a plurality of distinct vehicles. Each sub-planfor each vehicle of the plurality of distinct vehicles corresponds to aportion of the combined plan. In some embodiments, the sub-plans of thevehicles at least partially overlap with each other. In someembodiments, each sub-plan for each vehicle is distinct from thesub-plans of the other vehicles.

The planner 1908 is a service that interfaces with the router 1902 andsolver 1910 to service plan request and updates (e.g., additions ofpick-up/drop-off requests to existing routes, as described below). Insome embodiments, the planner 1908 creates cost matrices for each of theplurality of FRP problem instances generated by the grouper 1906. Insome embodiments, the cost matrices are generated from real-time ETAs(and, optionally, historical/predictive ETAs). The planner 1908 pipesthe plurality of FRP problem instances, with their cost matrices, intothe solver 1910 to solve.

The solver 1910 solves FRP problem instances (e.g., the plurality of FRPproblem instances passed to the solver from the planner). In someembodiments, the solver 1910 solves the FRP problem instances asinstances of the dial-a-ride-problem (DARP), which is a formulation ofthe fleet routing problem.

To that end, in some embodiments, the problem is broken down from ageneralized dynamic programming implementation into a graph searchproblem, and solved using an A* with an admissible heuristic orBidirectional Dijkstra's (for the non-time window variant). In someembodiments, states are broken down into the following representation:

s=(L ₁ , . . . , L _(M) , k ₁ , . . . , k _(N))

L _(i) ϵ{0, 1, . . . , N, N+1, . . . , 2N, 2N+1}

k _(i) ϵ{W, V _(j) , D}

M=|V|

where N is the number of vehicle drop-off pairs. An L state represents avehicle location: L=0 indicates the vehicle is in an unbounded state,0<L≤N indicates a pickup location (e.g., L=i is the pick-up state of thei^(th) vehicle when 0<i<N), and N+1≤L≤2N indicates a drop-off. The kstates represent the various passenger states, which are part of the set“waiting”, “in the jth vehicle”, or “dropped off”. Thus, s representsthe overall state of passengers and vehicles (e.g., the state of thesystem). In some embodiments, a termination state is a state in whichall k states are in the dropped off state. The number of total states fof the system can be calculated exactly as a function of the number ofrequests and the number of vehicles (0 is the “order” of the totalnumber of states):

f(M,N)=(2N+1)^(M)×(2+M)^(N)

O(M,N)=N ^(M) ×M ^(N)

Because of the exponential growth rate as a function of the number ofvehicles and number of passengers, some embodiments use exact optimalityalgorithms and keep requests to the solver limited. In some embodiments,however, capacity, time-based, Maximum-Position Shift (MPS), andconsistency constraints are used to help keep the graph search limitedand efficient for small problems (e.g., 2 vehicles with 5 pickup-dropoffpairs).

Graph Space Representation of the State of a Plurality of Passengers andVehicles

An FRP problem instance can be solved as an optimization problem of thefollowing form:

min ΣE_(V)(w ₁ ΣT _(i) +w ₂(αWT _(j))+(1−α)×RT _(j)))

Where Ti represent the traversal time of the legs of the route, WTjrepresents the total waiting time of the jth customer and RTj representsthe total ride time of the jth customer. This function can be minimizedand solved for the optimal routes by mapping our state space to a graphwith transition edges that incorporate individual elements in thisfunction. Alpha (α) is a constant that expresses a preference forpassengers being in the vehicle rather than waiting.

In some embodiments, the graph search is initialized from s₀=(0, 0, . .. , 0, W, W, . . . , W), and terminated when the terminating node isreached: S_(T)=(2N+1, 2N+1, . . . , 2N+1, D, D, . . . , D).

Consider, as an example, one vehicle with two pick-up/drop-offlocations. From the optimization function above we can calculate thetotal number of states as 45, however many of those will beinconsistent, e.g., (1, W, W). The graph for this example is shown inFIG. 24 (e.g., which shows a state graph representation of the pluralityof passengers and vehicles).

Here the first node 2400, “(0, W, W)”, indicates the initial state (avehicle in an unbounded position with no passengers), the nodes 2402indicate various states during the plan, and the nodes 2404-1 and 2404-2are terminal nodes. From a shortest path perspective, the problem is toroute from the first node 2400 to either of the terminal nodes 2404-1 or2404-2 (a one-to-any routing problem), where an edge from state s_(i) tos_(j) is given edge weights:

w(s _(i) , s _(j))=w ₁ ×t(s _(i) , s _(j))+w ₂×(α×x _(W)+(1−α)×X _(V))

Here the t function indicates the travel time to go from to S_(j), X_(W)indicates how many passengers are waiting in the state Si, and Xvindicates how many riders are in the vehicle in S_(i). Note that a sumof edge weights for a path exactly equals the minimization function, andso finding the shortest path through this graph is equivalent to solvingthe optimization problem.

Also note that the effective search space size (13 nodes) issignificantly smaller than the total number of states (45). A largenumber of states are inconsistent, and may not be practically reachable,which allows the graph to be traversed in a narrow subspace for thegraph. In some circumstances, the size of the subspace for the graphasymptotically approaches ⅓ of the size of the total space. Addingconstraints like capacity or MPS narrows the subspace even further,rather than increasing the search space as they might in a general MILPor CP solver.

Search Algorithms

Because some embodiments solve the DARP in graph space, some embodimentsapply graph search techniques and algorithms to speed up the search. Forexample, instead of applying a breadth first search, some embodimentsDijkstra's algorithm to speed up the search. For non-time-windowsearches, some embodiments search in the reverse direction and implementBidirectional Dijkstra's algorithm to resolve the search more quickly.

Even with time windows however, some embodiments apply a unidirectionalsearch in the forward direction with an admissible lower bound heuristicand use the A* search algorithm. Furthermore, any heuristics can becombined via a max function to find the best heuristic during thesearch. Below, exemplary heuristics are discussed.

Simple Lowest Bound Heuristic

A cost from a state SI to a terminal state ST can be written as follows:

C(S _(i) , S _(T))=Σ_(V)(w ₁Σ_(P) T _(i) +w ₂Σ_(P)(α×WT _(j)+(1−α)×RT_(j)))

Where the inner summations are over the shortest path P from state SI toterminal state S_(T). Because w₁ and w₂ are nonnegative, as well as theinner summands, and combined with the triangle inequality for routingdistances, a lower bound of the cost can be established:

Σ_(V)(w ₁Σ_(P) T _(i) +w ₂Σ_(P)(α×WT _(j)+(1−α)×RT _(j)))≥w ₁ ×t(S _(i), S _(T))=h(S _(i) , S _(T))

An A* heuristic function, hs, can then be defined as a minimum over alltermination nodes:

h _(S(s) _(i) ₎=min h(S _(i) , S _(T))

Pickup-Dropoff Bound Heuristic

A heuristic can be constructed taking into account information about thecurrent state's number of waiting, picked up, and dropped offpassengers. To that end, in some embodiments, the following values aredefined:

t _(P)=min t(S _(D) , S _(P))

t _(D)=min t(S _(P) , S _(D))

Where S_(D) are the drop-off states and sp are the pickup states. If avehicle is in a drop off currently and needs to make another pickupbefore being done, a lower bound of the cost can be established:

h _(D)(S _(D))=w ₁ ×t _(D)

Similarly, if there is a need for a pickup and a need to drop off atleast one person a lower bound of the cost can be established:

h _(P)(S _(P))=w ₁ ×t _(P)

Dynamic Requests

For large problems (e.g., 10 or more pickup requests and a fleet ofvehicles), a graph space exploration becomes quickly infeasible asrequests cross an acceptability threshold (e.g., the non-real timelatency threshold). For these cases, in some embodiments, the solversupports incremental plan updates, where it takes an existing solutionand inserts new requests into an existing path in the state space,instead of running an entire graph exploration.

Greedy Incremental Update

Some implementations insert a new pickup or drop-off into the existingpath by minimizing the change in time when inserting that new point.Some embodiments find the optimal insertion point p by solving thefollowing:

p=argmin c(s _(i) , p)+c(p, s+1)−c(s _(i) , s+1)

Where the s_(i) refers to the ith state in the solution, and c(s_(i),s_(j)) is the cost to go from s_(i) to s_(j). For example, FIGS. 25A and25B illustrate adding an insertion point, triangle 2500, to an existingpath.

Suppose there is an existing route with pickup/dropoff points and a needto service the triangle 2500 (e.g., pick-up the triangle at its currentlocation and drop it off somewhere else). Some embodiments calculatewhere to insert the triangle 2500 (e.g., which leg of the path to break)by minimizing the difference in time gained by breaking the leg. Notethat the time change should always be positive because of the triangleinequality.

ROUTING METHOD FOR A RIDE-SHARE TRANSPORTATION NETWORK

FIGS. 26A-26C are a flow diagram illustrating a routing method 2600 fora ride-share transportation network, in accordance with someembodiments. The method 2600 is performed at an electronic device (e.g.,vehicle routing server 1420, FIG. 14 , which may use the architectureshown in FIG. 19 ). While the following discussion describesimplementations where the vehicle routing server 1420 performs the stepsof the method 2600, in other embodiments, one or more of the steps areperformed by another electronic device, such as autonomous vehicle1410-2 (e.g., a computer system on vehicle 1410-2 with non-transitorymemory storing instructions for executing the method and one or moreprocessors for executing the instructions). Moreover, some operations inmethod 2600 are, optionally, combined and/or the order of someoperations is, optionally, changed. In some embodiments, some or all ofthe operations of method 2600 are performed automatically, without humanintervention.

As described below, method 2600 provides technical solutions to thefleet routing problem, discussed above. In particular, method 2600addresses the problem of how to route a fleet of vehicles to fulfillride requests that can be scheduled in advance, requested in real-time,or a combination of the two. More efficiently routing a fleet ofvehicles (e.g., a plurality of vehicles) to fulfill ride requestsconserves resources (e.g., cuts down on carbon emissions and/or batteryusage from autonomous vehicles), reduces traffic, and increasespassenger comfort and safety.

The vehicle routing server 1420 stores (2602) representations of aplurality of first passengers. Each of the representations of theplurality of first passengers includes a requested pick-up location anda requested drop-off location for a respective first passenger of theplurality of first passengers.

In some embodiments, prior to storing the representations of theplurality of first passengers, the vehicle routing server 1420 receivesa ride request from each of the plurality of first passengers. Therequest includes a requested pick-up location and a requested drop-offlocation for a respective first passenger. In some embodiments, the riderequest from each of the plurality of first passengers is a request toschedule a ride and includes a requested pick-up and/or a requesteddrop-off time. For example, in some embodiments, the ride request fromeach of the plurality of first passengers is received at least an hourbefore the requested pick-up and/or requested drop-off time (e.g., insome circumstances, the ride requests from at least some of the firstpassengers are received the night before the rides are needed). Forexample, in some embodiments, as described below, the ride requests fromthe plurality of first passengers are clustered. To that end, in someembodiments, the vehicle routing server 1420 separately clusters batchesof scheduled requests that fall within a predefined pick-up or drop-offtime window (e.g., the vehicle routing server 1420 separately batchesscheduled ride requests for pick-ups within each half-hour window).

In some embodiments, the ride request from each of the plurality offirst passengers is a real-time request (sometimes referred to as adynamic request) (e.g., a request to be picked-up and/or dropped off assoon as possible). In some embodiments, the ride requests from theplurality of first passengers are dynamic requests that are receivednear each other in time. For example, in some embodiments, as describedbelow, the ride requests from the plurality of first passengers areclustered. To that end, in some embodiments, the vehicle routing server1420 separately clusters batches of dynamic requests of a predefinednumber (e.g., 20 requests, 50 requests, 100 requests). In someembodiments, the vehicle routing server 1420 separately clusters batchesof dynamic requests that are received within a predefined time window(e.g., 1 second, 5 seconds, 30 seconds, 1 minute).

The vehicle routing server 1420 stores (2604) representations of aplurality of fleet vehicles. In some embodiments, the fleet vehiclesinclude (2606) a plurality of autonomous vehicles (e.g., autonomousvehicles 1410-2). Routing a fleet of autonomous vehicles solves atechnical problem because, unlike human-driven vehicles, autonomousvehicles cannot route themselves and thus must (at least in somecircumstances) be routed by a computer. In some embodiments, theplurality of fleet vehicles is (2608) operated by a single operator(e.g., Uber®, Ford®, or Waymo®). Routing a fleet of vehicles operated bya single operator solves a technical problem of distributing costs overthe fleet rather than an individual vehicle, which conserves resources(e.g., cuts down on carbon emissions and/or battery usage fromautonomous vehicles), reduces traffic, and increases passenger comfortand safety. Here, the term “costs” refers to cost model costs, which aredescribed throughout this document (e.g., with reference to method 1500,FIGS. 15A-5C).

In some embodiments, the vehicle routing server 1420 receivesinformation about the fleet vehicles (e.g., from the vehicles themselvesor from the operator of the vehicles). In some embodiments, theinformation about the fleet vehicles includes a location of each vehiclein the fleet. In some embodiments, the information about the fleetvehicles includes a number (e.g., count) of vehicles in the fleet. Insome embodiments, the information about the fleet vehicles includescapabilities of each vehicle (or at least a subset of vehicles) in theplurality of fleet vehicles, which can be used for routing the fleetvehicles in accordance with their capabilities as described withreference to method 1600, FIG. 16 .

The vehicle routing server 1420 generates (2610) a graph representationof a geographic map that includes the requested pick-up locations anddrop-off locations for the plurality of first passengers. As usedherein, the term “graph representation” means a set of nodes and edges.In some embodiments, the edges have costs (also referred to as weights).In some embodiments, the graph representation of the geographic mapincludes edges that represent roads. In some embodiments, as describedwith reference to method 1700, FIG. 17 , the graph representation of thegeographic map includes edges that represent paths through anintersection.

The vehicle routing server 1420 generates (2612) a state graphrepresentation of the plurality of first passengers and the fleetvehicles (e.g., as shown and described with reference to FIG. 24 ). Thestate graph representation of the plurality of first passengers and thefleet vehicles is distinct and separate from the graph representation ofthe geographic map. The state graph representation includes a pluralityof nodes connected by edges.

Each node of the plurality of nodes of the state graph representationrepresents a candidate state of the plurality of first passengers andthe fleet vehicles. A candidate state is a state that the plurality offirst passengers and the fleet of vehicles will acquire if a route isselected through that candidate state (e.g., a candidate state is apotential state that will become a selected state if a route through thepotential state is selected). For example, a candidate state may includeinformation indicating that passenger p_(j) is riding in vehicle v_(i).

A respective edge (e.g., each respective edge) of the state graphrepresentation represents an action of a respective vehicle picking upor dropping off a passenger. For example, an edge may represent vehiclev_(i) traveling to pick up passenger p_(j). Thus, a downstream nodeconnected to that edge will include information indicating thatpassenger p_(j) is riding in vehicle v_(i).

The respective edge has a cost that is based at least in part ontraversal of the graph representation of the geographic map. Forexample, the respective edge has a cost associated with the action ofthe respective vehicle picking up or dropping off the passenger, whichincludes a travel time cost or any of the other costs described in thisdocument (e.g., the costs described with reference to method 1500, FIGS.15A-15B). Thus, in some embodiments, some or all of the edges of thestate graph representation of the plurality of first passengers and thefleet vehicles represent a solution to an individual vehicle routingproblem (e.g., using the graph representation of the map) associatedwith the action of picking up or dropping off the passenger.

The vehicle routing server 1420 generates (2614) a set of routes for thefleet vehicles, including assigning each passenger of the plurality offirst passengers to be picked up and dropped off by a respective vehicleof the fleet vehicles (e.g., assigning a respective vehicle to eachpassenger, such that the respective vehicle picks up and drops off thepassenger). The routes for the fleet vehicles are generated byperforming a graph search of the state graph representation byevaluating a cost model that includes the costs of the respective edges.In some embodiments, generating the route for the respective vehicleincludes generating a route for the respective vehicle to an assignedpick-up location for the passenger (e.g., either the requested pick-uplocation or a different pick-up location that is near the requestedpick-up location). In some embodiments, generating the set of routesincludes generating a route for the respective fleet vehicle from theassigned pick-up location for the passenger to an assigned drop-offlocation (e.g., either the requested drop-off location or a differentdrop-off location that is near the requested drop-off location).

In some embodiments, the graph search is (2616) a bi-directional search(e.g., as described elsewhere in this document). Using a bi-directionalsearch solves a technical problem by allowing the graph search to becompleted more quickly. In some circumstances, using a bi-directionalsearch makes the use of a state graph representation practicable (e.g.,fast enough to be performed with real-time requests), thus giving riseto the technical benefits of method 2600 more generally (e.g.,conservation of resources, reduction of traffic, and increased passengercomfort and safety).

In some embodiments, the state graph representation is generated whileperforming the graph search of the state graph representation. Forexample, the vehicle routing server 1420 generates an initial set ofnodes and edges of the state graph representation. As described ingreater detail below, in some embodiments, the vehicle routing server1420 uses a lower-bound heuristic to eliminate portions of the stategraph representation. Thus, in some embodiments, the vehicle routingserver 1420 forgoes generating some or all of the nodes and edges forportions of the state graph representation that have been rendered mootby the lower bound heuristic (e.g., forgoes generating all of the edgesand nodes rendered moot by the lower bound heuristic except for thoseneeded to calculate the lower bound heuristic in the first place).

In some embodiments, as described with reference to FIGS. 19-23 ,generating the set of routes for the fleet vehicles includes (2618):assigning each passenger of the first plurality of passengers to one ormore candidate pick-up locations based on the first passenger'srequested pick-up location; assigning each passenger of the firstplurality of passengers to one or more candidate drop-off locationsbased on the first passenger's requested drop-off location; clusteringthe first plurality of passengers according to their assigned candidatepick-up locations and drop-off locations; assigning respective vehiclesof the fleet vehicles to a plurality of clusters; and parallelizing therouting of the fleet vehicles according the assigned plurality ofclusters for the respective vehicles of the fleet vehicles. In someembodiments, the candidate pick-up (drop-off) locations are permissiblepick-up (drop-off) locations near the first passenger's requestedpick-up location. In some embodiments, the candidate pick-up (drop-off)locations are locations that an autonomous vehicle can access (i.e.,accessible locations for an autonomous vehicle). In some embodiments,the vehicle routing server 1420 provides information to the respectivepassengers directing the respective passengers to their assigned pick-uplocations (e.g., informs the respective passengers of their assignedpick-up locations, or routes the respective passengers to walk to theirassigned pick-up locations).

In some circumstances, the clustering process described above allows thefleet routing problem to be parallelized in an efficient manner. In somecircumstances, the clustering process described above allows method 2600to be completed more quickly. In some circumstances, the clusteringprocess described above makes the use of a state graph representationpracticable (e.g., fast enough to be performed with real-time requests),thus giving rise to the technical benefits of method 2600 more generally(e.g., conservation of resources, reduction of traffic, and increasedpassenger comfort and safety).

In some embodiments, as described with reference to the sections labeled“Simple Lowest Bound Heuristic” and “Pickup-Dropoff Bound Heuristic,”performing the graph search includes (2620) limiting a number of statesexplored in the state graph representation using a lowest-boundheuristic by forgoing subsequent exploration from any node in the stategraph representation for which a lowest-bound of the cost functionexceeds a cost of an already-determined set of routes for the fleetvehicles. As used herein, a lower-bound heuristic is a condition thatlimits a search space (e.g., limits exploration of the state graphrepresentation by forgoing subsequent exploration from a respectivenode) based on determination of a lower-bound of the cost function. Alower-bound of the cost function for a respective node means that anyactual route from the respective node will have a cost that is greaterthan or equal to the lower-bound of the cost function (e.g., thelower-bound of the cost function is a floor of the cost of a route fromthe respective node). Thus, in some embodiments, the vehicle routingserver 1420 determines a lower-bound for a respective node (e.g., aplurality of nodes) in the state graph representation and forgoessubsequent exploration from the respective node (e.g., the plurality ofnodes) in the state graph representation based on the lower-boundheuristic (e.g., if the lower-bound on the cost exceeds the costs of analready-existing solution to the routing problem).

For example, if the graph exploration goes to a respective node thatrepresents a passenger entering a respective vehicle, at a minimum, thevehicle must drop-off that passenger. If the cost of dropping off thatpassenger results in a cost that exceeds the costs of an alreadyexisting solution to the routing problem, the vehicle routing system1420 forgoes exploring anything downstream of the respective node (e.g.,forgoes generating all or some of the graph downstream of the respectivenode). In some circumstances, eliminating portions of the state graphrepresentation from exploration allows method 2600 to be completed morequickly. In some circumstances, eliminating portions of the state graphrepresentation makes the use of a state graph representation practicable(e.g., fast enough to be performed with real-time requests), thus givingrise to the technical benefits of method 2600 more generally (e.g.,conservation of resources, reduction of traffic, and increased passengercomfort and safety).

In some embodiments, the cost model includes (2622) pick-up wait timesfor the first passengers. In some embodiments, the cost model includes acost of a passenger waiting for a vehicle and/or a cost of a passengerwaiting in a vehicle (e.g., during the ride). In some embodiments, thepassengers need not be human (e.g., may be parcels), and thus the costmodel does not include a cost of a passenger waiting for a vehicleand/or a cost of a passenger waiting in a vehicle. Instead, for example,when the passengers are parcels, the cost model is weighted heavilytoward an on-time delivery. In some embodiments, the passengers comprisea combination of human riders and parcels, the request identifies thepassenger as either human or a parcel. When the passenger is human, thecost model includes a cost of the passenger waiting for a vehicle and/ora cost of the passenger waiting in a vehicle. When the passenger is nothuman, the cost model does not include a cost of the passenger waitingfor a vehicle and/or a cost of the passenger waiting in a vehicle.

The vehicle routing server 1420 routes (2624) the fleet vehicles inaccordance with the generated set of routes. In some embodiments,routing the fleet vehicles in accordance with the generated set ofroutes includes sending each vehicle's route to the vehicle. In someembodiments, routing the fleet vehicles in accordance with the generatedset of routes includes sending the generated set of routes to theoperator of the fleet vehicles. In some embodiments, routing the fleetvehicles in accordance with the generated routes includes routing arespective vehicle assigned to pick up a passenger to an assignedpick-up location for the passenger (e.g., either the requested pick-uplocation or a different pick-up location that is near the requestedpick-up location). In some embodiments, routing the fleet vehiclesincludes routing the respective vehicle from the assigned pick-uplocation for the passenger to an assigned drop-off location (e.g.,either the requested drop-off location or a different drop-off locationthat is near the requested drop-off location).

Operations 2626-2632 describe a method of inserting an additionalpassenger into the generated set of routes. Inserting an additionalpassenger into the generated set of routes allows the fleet routingproblem to continue to be solved efficiently after the set of routes hasalready been generated, which conserves resources (e.g., cuts down oncarbon emissions and/or battery usage from autonomous vehicles), reducestraffic, and increases passenger comfort and safety.

In some embodiments, the vehicle routing server 1420 receives (2626) aride request from a second passenger, distinct from the plurality offirst passengers. The request includes a requested pick-up location anda requested drop-off location for the second passenger. In someembodiments, the request from the second passenger is a dynamic request(e.g., a real-time request to be picked up and dropped-off as soon aspossible). In some embodiments, the request from the second passenger isreceived after receiving the requests from the plurality of firstpassengers. In some embodiments, the request from the second passengeris received after the set of routes is generated. In some embodiments,the request from the second passenger is received while some or all ofthe vehicles are performing their respective routes (on their respectiveroutes) of the set of generated routes.

In some embodiments, the vehicle routing server 1420 inserts the secondpassenger's request into the existing generated set of routes. To thatend, in some embodiments, the vehicle routing server 1420 updates (2628)the route for a respective vehicle of the fleet vehicles, includingassigning the second passenger to be picked up and dropped off by therespective vehicle, in accordance with one or moreleast-expensive-insertion criteria. For example, rather than performingthe state graph search from scratch with a new cost model for the fleetvehicles, the first plurality of passengers, and the second passenger,the vehicle routing server 1420 determines which existing route is theleast expensive route to insert the second passenger's request accordingto the new cost model for the fleet vehicles, the first plurality ofpassengers, and the second passenger. In some embodiments, as describedbelow, a limited number (e.g., predefined number) of “trading”passengers is permitted (e.g., trading one passenger).

In some embodiments, updating the route for the respective vehicle ofthe fleet vehicles includes (2630) inserting a pick-up location and adrop-off location for the second passenger into an existing route forthe respective vehicle without modifying pick-up and drop-off locationsfor passengers already assigned to the respective vehicle (e.g., but notyet picked up). In some embodiments, updating the route for therespective vehicle of the fleet vehicles includes (2632): inserting apick-up location and a drop-off location for the second passenger intoan existing route for the respective vehicle; and reassigning apassenger already assigned to the respective vehicle (e.g., but not yetpicked up) to a different vehicle of the fleet vehicles (e.g., tradingpassengers).

In some embodiments, updating the route for a respective vehicle of thefleet vehicles, including assigning the second passenger to be picked upand dropped off by the respective vehicle, by: assigning the secondpassenger to an existing cluster (e.g., generated at operation 2618);and updating the route for the vehicle assigned to the existing cluster.

In some embodiments, because the complexity of the state graphrepresentation goes up as more requests come in, updating the route fora respective vehicle of the fleet vehicles includes updating the stategraph representation (e.g., such that the state graph representation isa representation of the plurality of first passengers and the secondpassenger, as well as the fleet of vehicles) and performing a graphsearch using at least a portion of the updated state graphrepresentation.

It will also be understood that, although the terms “first,” “second,”etc. are, in some circumstances, used herein to describe variouselements, these elements should not be limited by these terms. Theseterms are only used to distinguish one element from another. Forexample, a first vehicle could be termed a second vehicle, and,similarly, a second vehicle could be termed a first vehicle, whichchanging the meaning of the description, so long as all occurrences ofthe “first vehicle” are renamed consistently and all occurrences of thesecond vehicle are renamed consistently. The first vehicle and thesecond vehicle are both vehicles, but they are not the same vehicle(e.g., the second vehicle is an additional vehicle).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the claims. Asused in the description of the embodiments and the appended claims, thesingular forms “a”, “an” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willalso be understood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in accordance with adetermination” or “in response to detecting,” that a stated conditionprecedent is true, depending on the context. Similarly, the phrase “ifit is determined (that a stated condition precedent is true)” or “if (astated condition precedent is true)” or “when (a stated conditionprecedent is true)” is, optionally, construed to mean “upon determining”or “in response to determining” or “in accordance with a determination”or “upon detecting” or “in response to detecting” that the statedcondition precedent is true, depending on the context.

The foregoing description included example systems, methods, techniques,instruction sequences, and computing machine program products thatembody illustrative embodiments. For purposes of explanation, numerousspecific details were set forth in order to provide an understanding ofvarious embodiments of the inventive subject matter. It will be evident,however, to those skilled in the art that embodiments of the subjectmatter are, optionally, practiced without these specific details. Ingeneral, well-known instruction instances, protocols, structures andtechniques have not been shown in detail.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the embodiments to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples and their practical applications, to thereby enable othersskilled in the art to best use the embodiments and various embodimentswith various modifications as are suited to the particular usecontemplated.

What is claimed is:
 1. A method of routing an autonomous vehicle,comprising: at a computer system including one or more processors andmemory: receiving information obtained from a camera on a second vehicledistinct from the autonomous vehicle; automatically identifying a roadcondition using image analysis of the information received from thecamera on the second vehicle; receiving a request to route theautonomous vehicle from a first location to a second location; and inresponse to the request: generating a cost model for routing theautonomous vehicle, wherein the cost model includes a cost of the roadcondition automatically identified from the information received fromthe camera on the second vehicle; selecting a route from the firstlocation to the second location in accordance with the cost model; androuting an autonomous vehicle in accordance with the selected route.